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INTRODUCTION 


C ity-led efforts to build coordinated systems of afterschool programming are an 
important strategy for improving the health, safety and academic preparedness of 
children and youth. Over the past decade, municipal leaders, foundations, major 
nonprofit intermediaries, and school and community-based providers have increasingly come 
together to expand the number of high-quality programs available, increase youth participation, 
and improve outcomes for young people. Yet even cities with strong leadership and effective 
coordinating entities are often challenged by the lack of reliable information to answer basic 
questions about the scope and impact of afterschool programs in their communities. 

To improve access to critical information - for city leaders, funders, program directors and 
front-line staff - cities need both the technology to track and correlate information on youth 
participation across dozens of organizations, and networks of skilled professionals to share, 
analyze, and act on that information. In most cities, these systems do not yet exist or are only 
partially complete. 

The decision to build or enhance a management information system (MIS) raises its own set 
of tough questions about what information to collect and how to use it; how to negotiate 
data sharing agreements without violating privacy laws; how to think about the difference 
between evaluating youth outcomes and measuring program quality; and whether to build 
or buy the technology backbone that will support the data needs of policymakers, service 
providers, program managers, and researchers. 

The National League of Cities (NLC), through its Institute for Youth, Education and Families, 
produced this report to help city leaders, senior municipal staff and their local partners 
answer those questions as they work to strengthen and coordinate services for youth and 
families, particularly for those cities building comprehensive afterschool systems. It provides 
a thorough review of what it takes to build effective management information systems for 
youth services, including a detailed “how to” guide for preparing for and implementing an 
afterschool MIS and strategies for addressing privacy and security concerns in collaboration 
with schools and families. 

The report concludes with a comparative analysis of six leading commercial MIS vendors and 
a cost calculator to help communities explore the expense of differently configured systems. 
In addition, city leaders and staff will find a growing library of resources on the NLC website 
to speed their progress, including sample requests for proposals, data sharing agreements, 
system architectures, and other useful tools to borrow and adapt. 


WHY BUILD AN AFTERSCHOOL MANAGEMENT INFORMATION SYSTEM? 


An abundance of research has demonstrated the value of high-quality afterschool programs 
and substantiated their positive influence on the health, safety, school attendance, and 
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academic performance of youth. Yet it is an immense management challenge for cities and 
other stakeholders to know if they are fully meeting the needs of youth and maximizing the 
potential impact of high-quality afterschool programming. 

At its best, a strong, multi-faceted afterschool MI system can improve youth outcomes by: 

• Providing policymakers and funders with accurate information on the utilization, 
quality and impact of afterschool programs to make better decisions and targeted 
investments at the systems level; 

• Offering regular feedback to program managers and staff about the effectiveness of their 
efforts, both in absolute terms and relative to other programs, to promote continuous 
improvement; 

• Reducing the time and money 
that programs spend complet- 
ing paperwork to meet report- 
ing requirements, freeing up 
valuable resources for direct 

; and 

• Empowering program sites 
and instructors with (near) 
real-time information on stu- 
dent outcomes such as atten- 
dance, behavior and academic 
performance that allows sites 
to tailor their instruction more 
closely to the needs of the 
youth they serve. 

Yet as the RAND Corporation described in its authoritative review of The Wallace Foundations 
first round of investment in citywide afterschool systems, this level of coordination and 
intentional use of integrated data is rare. 

Within cities, out-of-school time (OST) provision can be fragmented and uncoordinated. 

Providers rely on an unsteady and often insufficient patchwork of city, state, federal and 

private funding and user fees. Further, in many cities, public funding is funneled through 

a variety of youth-serving agencies without interagency coordination. 

^ RAND, The Power of Data to Improve After-School Programs Citywide 

The experience of the two dozen city leaders surveyed and interviewed by NFC for this 
report confirmed the RAND reports observation. Collectively, these cities had patched 
together funding from general revenue, federal 21st Century Community Fearning Center 
grants, local and national foundations, state awards, city tax levies, and the United Way to 
support and expand the availability of OST programs for students. Each of these sources 
of support has its own application requirements, funding cycles, reporting demands, and 
eligibility standards (Chart 1 on page 3). 


programming with youth 


We've come to the realization that the 
better coordinated the data collection effort 
is, tne better we can serve Bridgeport's 
children. They desperately need additional 
resources to address academic deficiencies. 

We need to step up efforts to obtain this data 
and use it to make informed decisions about 
the types of programs and services 
our children will receive. " 

~ Tommy Popo , Lighthouse Progrom Director, City of Bridgeport, Conn. 
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Chart 1 - What funds city ost coordinating entities 
[«-24] 
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Chart 2 - Significant ost providers 
[n-24] 
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Source: 2011-2012 survey of city leaders by NLC 


This funding supports programs at dozens of nonprofit and community-based programs, 
schools, city recreation departments and libraries, faith-based providers and others. Each of 
these providers has its own mission and approach to quality, and each negotiates a separate 
set of reporting and compliance demands with its funders (Chart 2). 

City leaders strive to coordinate the millions of dollars being spent within their cities on OST 
programs to ensure they are consistently of high quality and reasonably well aligned with 
other initiatives underway. In many cities, major progress has been made toward measuring 
and improving the return on these investments. 

However, many challenges remain. City leaders continue to struggle in assessing the reach and 
impact of their afterschool systems. How many youth have access to afterschool programs? 
How many of those with at least nominal access actually attend? Which of these programs 
are of high quality and have the impact that funders, providers and parents would hope for 
and expect? These are fundamental questions that, without an afterschool MIS, city leaders 
find it difficult or impossible to answer. 

Only 27 percent of the cities that responded to a 2011 NLC survey believe they have a 
reasonably accurate count of how many youth are eligible to attend OST programs. It is 

not surprising, then, that city leaders identify “reliable information” about afterschool 
programming and impact as the area in which they desire the most technical assistance. 

What does “reliable information” on afterschool programs encompass for a city leader? It be- 
gins with an inventory of programs available in his or her community, information on the en- 
rollment and participation of youth in afterschool activities, data by which to assess whether 
programs are improving outcomes for youth, and responses from providers and parents on the 
many ancillary needs of youth in the system for tutoring, transportation, and social support. 
To be useful, this information must be accurate, timely, and reasonably comprehensive. Un- 
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WHAT IS A MANAGEMENT INFORMATION SYSTEM? 


Management Information Network + 

Management Information Technology = MIS 

An afterschool management information system is made up of a network of professionals 
who purposefully create, analyze and use information to improve youth development 
programs, and a stack of technologies that facilitate the work of these professionals. 
The most important decisions in building an Ml system are not which technology to 
use. It is more important to decide which people to include and how to connect them 
to this flow of information. 

Most fundamentally, Ml systems allow afterschool providers to record program 
activities, such as youth enrollment, attendance and participation, and to summarize 
these activities through fixed, regularly scheduled reports to agencies and funders. 

Ml systems are often designed to do much more than this: 

• Providers may use them to assess youth developmental assets, track the 
demographic characteristics of their participants, and support case 
management and referral. 

• Policymakers may use Ml systems to link afterschool participation with other 
administrative records to flag students who appear to be "at risk," estimate 
whether programs are improving college and career readiness, and evaluate 
the overall impact of a city's investment in out-of-school time programming. 

When analyzed in support of organizational efforts to improve compliance, 
professional development, and process improvement, the purpose of Ml systems is to 
create knowledge that enables providers to run more effective programs and helps 
administrators allocate resources to achieve their policy objectives most efficiently. 

Most afterschool Ml Systems are web-based. They may be custom-built by local school 
districts or city departments or purchased as a service from commercial vendors, 
including: 

Cayen Efforts-to-Outcomes (Social Solutions) 

CiviCore EZReports (ThomasKelly Software) 

COMET (SophiTEC) KidTrax (nFocus) 

CommunityTech Knowledge YouthServices.net (Cityspan) 

For more information on each of these vendors, see Section 4 of the report. 


fortunately, what city leaders more commonly receive is a mix of electronic and paper files of 
varying degrees of completeness and accuracy, covering a limited set of city-funded providers. 

According to NLC’s 2011 survey: 

• Only 58 percent of cities used a management information system. 

• Those who did not use a MIS were unsatisfied with their process for tracking information, 
and believed it was inadequate to support their OST systems-building work. 
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• Only 38 percent of cities share information with their schools either “regularly” or 
“frequently”; another 38 percent never receive information from their schools. 

However, city leaders are quickly overhauling these outmoded arrangements, moving away 
from cumbersome paper records to capturing attendance online, establishing benchmarks 
against which to measure the results of their investment in OST, and building relationships 
with schools to share information on youth outcomes - in some cases, in real time. 


STRUCTURE OF THE REPORT 


This report describes a number of the most promising approaches to building afterschool 
management information systems. It is a compendium of “what works,” containing numerous 
examples of efforts led by different city departments, nonprofit intermediaries, schools and 
foundations. These examples include both self-built and commercial systems. Several of the 
systems profiled here are being developed as comprehensive databases of youth services while 
others are narrowly targeted to participation in afterschool programs. 

The report is divided into four sections that comprise a practical roadmap for city leaders 
seeking to develop or improve an MI system to support their local afterschool initiatives. The 
first section outlines four key benefits of using data to guide afterschool investments. Section 
two presents an MIS “readiness checklist” that city leaders can use to build a management 
information system that responds to local needs and priorities. Section three offers guidance 
on navigating the data privacy and security issues that must be considered when handling and 
sharing sensitive information on youth. The final section compares the MI system features 
offered by the top commercial vendors that are working with cities to manage information 
on local afterschool programs and other youth services. 
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SECTION 1 : 


FOUR HIGH VALUE USES 
OF AFTERSCHOOL DATA 


M anagement information systems (MIS) should be at the center of citywide efforts 
to collect, store, link, analyze, report, learn from, and use information on the 
afterschool programs that cities and their local partners operate. 

The diagram below illustrates how an MIS can support data-driven decision making as 
information flows back and forth across the site, agency, coordinating entity, local afterschool 
network, and funder levels. As the information collected in an MIS travels from a program 
to its funders - including the elected officials who allocate local revenues to city, school- 
based or nonprofit programs - it is used to evaluate youth outcomes and system impact, 
to demonstrate return on investment and allocate scarce afterschool resources, and to 
sustain and expand support for high-quality afterschool programs. In a high functioning 
afterschool network, a systems coordinating entity “closes the loop” by sending the results 
of these analyses back down to individual providers and sites, using them to facilitate peer 
benchmarking and empower afterschool providers with more complete information on the 
youth they serve. Each of these “high value” uses of afterschool data is described in more 
detail below. 


Afterschool 

Information 

Flow 


OST NETWORK, 
FUNDERS & THE PUBLIC 


feedback 

& 

improvement 



external inputs: 
data & expertise 


I r program 
evaluation 
and policy 
analysis 


Four High Values Uses 

1 Assess youth outcomes 
and system impact 


j§f§ Promote accountability, 
demonstrate return on 
investment, and allocate 
scarce resources 

3 Empower OST providers 
and reduce paperwork 

4 Facilitate peer bench- 
marking and profes- 
sional development 
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1 . ASSESS YOUTH OUTCOMES AND SYSTEM IMPACT 


Among the most important uses of an MI system is to provide a city’s coordinating entity 
with the kind of information on system usage and youth outcomes that can inform decisions 
about efforts to expand access, improve program quality, and link afterschool initiatives to 
broader citywide goals such as college and career readiness and public safety. Measuring impact 
begins with answering basic questions about the scope of afterschool services available in the 
community, counting the number of participants in different programs, assessing how many 
youth are eligible to participate, and evaluating what proportion of those youth are enrolled 
in high-quality programs. Ideally, it also involves either directly measuring developmental 
assets or integrating data with outside systems to evaluate how afterschool programs are 
influencing a broader set of outcomes related to academic achievement, employment or public 

safety. Mapping these results can also highlight areas of 
opportunity for further afterschool systems development. 

According to city afterschool leaders who responded to 
NLC’s 2011 survey, only 27 percent of cities believed 
that they had a “fairly exact” count of how many youth 
are eligible to attend afterschool programs. Cities esti- 
mated that as many as 60 percent of those youth who do participate are attending pro- 
grams that do not receive funding or professional development from the city. 

Once obtained, the information on afterschool eligibility and participation may be surprising. 
One community discovered, upon implementing an MI system that gave each child a swipe 
card to check in to afterschool programs automatically, that actual attendance at several sites 
was sharply less than the figures those providers had been reporting by email. 

City leaders are also interested in being able to tie their investments to specific youth 
outcomes and linking afterschool programs to broader citywide goals such as college and 
career readiness or public safety. Some cities use MI systems to measure youth development 
outcomes directly with pre- and post-tests such as the Search Institute’s Developmental Asset 
Profile (DAP). Many others compare the academic and behavioral outcomes of afterschool 
participants in school against those of non-participants over time, and are able to make 
specific claims about how their programs improve student attendance and grade completion. 

Just as it is a major missed opportunity for funders to collect afterschool program information 
without returning it to agencies as knowledge they can use to improve their programs, it is 
not enough for MI systems to facilitate peer learning without providing city leaders with 
the kind of aggregate information they can use to improve public policy. As one municipal 
director tartly observed of her city’s management system: 

I can’t say that this process is, shall we say, the best that it can be. I think that it’s great that 
we have providers hearing this and having an opportunity to visit with each other. But I 
am not pleased with the level of reporting out, so that policymakers and those that support 
these programs can see the data and use it.... It needs to be written up in a way that’s 
consumer-friendly, and the audience of people that should have this information should be 
so much broader. 


Thee 

shouk 


The audience of people that 
snould have this information 
should be so much broader. " 
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PROGRAM QUALITY ASSESSMENTS VERSUS YOUTH OUTCOME EVALUATIONS 


Program quality assessments and youth outcome evaluations are complementary 
approaches to understanding and improving afterschool programs. But they are very 
different tools with distinct uses and audiences, as Charles Smith, the executive director 
of the Weikart Center describes: 

There's almost a difference in ethic in terms of the way that the two very different pur- 
poses are viewed. One is evaluative - we want to come to some kind of summative 
judgment about the value of the services that the program is providing and we want 
to use that judgment in a higher-stakes way to allocate funds or assign professional 
development. And the other is this quality improvement side, where what we want 
to see is continuous improvement and to hold providers accountable to that process. 

Program quality assessments (PQA) are most concerned with, and therefore rele- 
vant to working with, afterschool providers . They utilize performance standards and mea- 
sures as professional development and can be an excellent way to initiate a conversation 
about quality improvement in programs that are skeptical of being benchmarked against 
youth outcomes that might be caused by many factors outside of the afterschool program. 

It is a weakness of program quality assessments that they are not directly linked to 
the system-level outcomes that are important to funders, elected officials, and other 
policymakers. Even those who find value in the PQA acknowledge that, as one city 
leader described it, "there's sort of this leap of logic that if your program is of high quality 
then you have good youth outcomes." But this proposition can be (and is) disputed, and 
PQA cannot, by itself, demonstrate to those that are skeptical of afterschool programs 
that they are receiving the desired return on their investment. 

Youth outcome evaluations are most directly useful at the systems level to identify 
areas of high and low performance, inform further investments, and sustain support for 
afterschool programs by demonstrating their value in supporting broader policy objec- 
tives and community priorities. Outcome evaluations can provide a more direct measure 
of what cities "really" care about: engaged, resilient, college and career-ready youth. 

Critics of youth outcome evaluations point out that this kind of measurement has a track 
record of disrupting the good work that it wishes to confirm. Management information sys- 
tems may emphasize compliance rather than improvement, and data reported to agency 
and system leaders by sites may never trickle back down to inform site-level practice. In 
other cases, funders may impose outcome measures for results that providers do not be- 
lieve they can reasonably be held accountable for achieving. 

In practice, many cities use both tools - and the most adroit use each to inform the other. 
In Nashville, Laura Hansen of Metro Nashville Public Schools reported that standardized 
outcome measures and performance data helped the process of allocating funding be 
more transparent: 

It can be scary, too. We've had some folks worried that this is going to be used as a 
stick to say 'hey, based on what we've seen, we don't want you back next year.' But 
our message to providers is that our goal is to use this as a continuous improvement 

(Continued) 
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(Continued from previous page) 


mechanism and, just like with [their program quality assessments], they'll get their 
program's feedback and they'll know if there's an issue, such as a high participant 


turnover rate, they'll have the opportunity to improve that. In marrying that to the 
quality standards, we can link professional development to what's going on at each 
program. We can take the YPQA coaches and send them to the programs where it 
looks like the students are really struggling. I think it can almost be an early warning 
system, and give people an opportunity to improve their program. The improvement 
part is what we're really after. 




Related to their value in gauging afterschool system- and program-level reach and impact, MI 
systems enable city leaders and private funders to hold programs accountable for achieving 
desired results and analyze at a policy level where their investment of afterschool dollars is 
most needed and is likely to bring the largest return. According to city responses to an NLC 
survey conducted in 201 1-12, the top reasons city leaders want to implement MI systems are 
to improve accountability to funders, demonstrate the value of investing in afterschool and 


more effective at driving youth outcomes than so-called “gym and swim” activities. Investment 


registration information, MI systems can also help city decision-makers map the capacity 



improve program quality (see chart below). 

CHART 3. City Leaders Want MIS to Improve Accountability and Increase Program Quality 


Improve accountability to funders and the public 
Demonstrate the value of afterschool 
Improve program quality 


Link in-school and out-of-school enrichment 


Evaluate program and provider quality 


Save time and money 




89 % 


Better target afterschool services 
Improve contract management 


| 62 % 
60 % 


50 % 55 % 60 % 65 % 70 % 75 % 80 % 85 % 90 % 95 % 

Source: 2011-2012 survey of city leaders by NLC 

One lesson many cities have drawn from the evidence is that high-quality programs are much 


in professional development to improve and refocus existing programs has followed. 

By collecting broad and accurate participation data that includes demographic and 


of their afterschool systems against their actual use and highlight areas of growing demand. 
Funds can be reallocated to underserved neighborhoods and critical age ranges. 
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The mayor has asked questions like, 'How many kids are in afterschool 
programs and are we missing any neighborhoods ?' Right now we can't 
answer those questions as a city department, never mind for the city as 
a whole. With the launch of this data system, we can now start to pull 
that information and look together at 'where are the gaps, where are the 
needs, where are we over- or under-serving?"' 

~ Catherine Penkert, City of Saint Paul , Minn. 

In addition, city leaders use hard data from MI systems to inform difficult and sometimes 
politically sensitive decisions to cut off funding to ineffective programs, thereby freeing up 
resources for programs that have a bigger impact. As one mayor’s advisor told NLC, to begin 
this conversation, policymakers need to bring clear quality standards and solid performance 
measures to the table: 

The mayor recognizes that most of these providers don’t have a consistent approach 
to quality standards. And he’s looking at my office to give him a sense of [what they 
realistically can be accountable for improving] , and to get enough information to make 
a decision about which of these programs that we have been operating forever should 
or should not continue. That’s going to be a very hard conversation to have, and you 
can’t do it anecdotally. 

As Priscilla Little, initiative manager at The Wallace Foundation, has said, “Given the 
new economic climate in which cities and states are operating, out-of-school time is an 
increasingly data-driven enterprise with results-based decision making. Evaluations are non- 
negotiable. You have to have that data to support your claims.” 

On the flip side, MI system data can help local leaders make the case for sustaining and 
expanding high-quality afterschool programs. “Accountability,” in this sense, is not only 
a bureaucratic necessity - it provides a framework through which afterschool providers 
and their supporters in local government can make a powerful argument for increased 
investment in afterschool systems. As Kim Luce, the 21 st Century Community Learning 
Center director for Buffalo Child and Adolescent Treatment Services, argues, accountability 
is about “getting people on board to see the benefit [of this work] , and to bring in national 
funding which would improve access and opportunity for quality programming. Anytime 
you are writing for support, if you don’t have good data, your case is weakened.” 

Irrespective of federal mandates, programs are often very highly motivated to demonstrate to 
elected officials and parents that their children and their dollars are being carefully managed. 
As Amy Phuong, the City of Atlanta’s Chief Service Officer, explained: 

Residents not only want us to make essential programs and services available, they want 
us to be accountable for making good investments toward high-quality programs. Mayor 
Reed understands this implicitly, and it’s my job to get him the data to demonstrate that 
this investment in Atlanta’s Centers of Hope is money well spent. 
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3. EMPOWER AFTERSCHOOL PROVIDERS AND REDUCE PAPERWORK 


Afterschool data collected in MI systems allow city and non-city program providers to make 
better decisions and work more efficiently in several ways: 


Informing Agencies and Managers 

Individual afterschool programs generate management information on enrollment, 
attendance, participant demographics, and often parent or student satisfaction. Rapid 
shifts in any of these factors can be a signal to managers to step in and provide more direct 
program oversight, find additional resources, or arrange professional development. Getting 
agencies this kind of “business intelligence” is crucial to maintaining high-quality programs. 
Providing instructors with information that is continuous and immediate, rather than semi- 
annually reporting the outcomes of students no longer enrolled in their programs, is crucial 
to achieving better outcomes for youth. 

Rob Clark, the afterschool director for the Family League of Baltimore City, sees an 
opportunity to combine information on student academics and program participation to 
identify students who would benefit from additional assistance: 

On the pro-active side - even starting in the spring and using data from the year just 
finishing - it would be great to take a look at those kids, to sit down with the principal 
and recruit for our summer learning program based on some of the data that we re seeing 
on which students need the extra help. 

These tools provide afterschool managers with the type of client information that is taken for 
granted by other professionals. They also transform the kind of engagement that is possible 
between agencies and programs, and between programs and parents, from a yearly “report 
card” to an ongoing, substantive conversation about improving results. 


Empowering Program Sites and Instructors 

One of the more obvious - but difficult to implement - advantages of an MI system is the 
potential for afterschool programs to view participating youths’ school records in (near) real 
time and to target and tailor their instruction accordingly. Changes in a student’s school 
attendance, behavior, and academic ranks are all important signals to afterschool program 
providers. 

Currently, many afterschool instructors receive this information only if they are school-based 
or fortunate to have a very close relationship with their students’ teachers and principal. 
Often, instructors receive a report with aggregate results for children six or 12 months after 
the end of the class. As Edwin Hernandez, who co-leads the pilot of Grand Rapids’ new 
“Believe 2 Become” initiative, points out, “It’s of no use to know the attendance rates of your 
children six months after you’ve had them in your system. We want this data to be made 
available to providers on the ground as they are serving kids.” For a description of the Believe 
2 Become MIS, see page 52. 
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Reducing Paperwork 

MI systems automate the creation of routine reports for instructors and site managers. The 
Providence After School Alliance (PAS A), for example, uses an MI system to create the roster 
that matches students leaving school with their assigned program and bus. Something as 
simple as this report can be the “killer app” that wins over skeptical providers, says PASAs 
quality consultant, Elizabeth Devaney. 

MI systems can also reduce or eliminate the demands on individual sites to gather student 
academic data and fill out compliance reports. In Philadelphia, Thomas Sheaffer, director 
of policy and evaluation within the Deputy Mayors Office for Health and Opportunity, 
is planning to link his department s MIS with the school district s academic records in the 
city’s data warehouse. Instead of each of his individual programs tracking its own student 
outcomes and filing compliance reports with multiple state and local funders, the city may 
be able to batch and send these reports in a fraction of the time. Philadelphia is already using 
a shared external evaluator to coordinate all of the school district and Archdiocesan student 
data required by the 21 st Century Community Learning Center data system, PPICS. 


FIVE DIFFERENT FLAVORS OF AFTERSCHOOL INFORMATION 


Ml systems can help afterschool leaders and providers collect and interpret information 
from multiple sources, including surveys of program quality and parent satisfaction, 
assessments of youth behavior and development, program participation information, 
and school transcripts. Each source has its own set of uses and limitations: 

Program participation (attendance) is the most fundamental element tracked 
by an afterschool MIS. In many cities, program attendance is linked directly to the fee 
paid to afterschool providers by city agencies and private funders. Attendance is also 
a useful, though inexact, proxy for program quality and client satisfaction: high-quality 
programs prioritize regular attendance by the youth they enroll while programs with 
poor reputations struggle to enroll and retain students. Use in M/5 : Ml systems employ 
a number of strategies to track youth participation, including online rosters, handheld 
mobile devices, and swipe cards. Cross-tabulating and aggregating this information 
for site managers, agency staff and city policymakers are among the most important 
features of an afterschool MIS. 

Program quality is usually measured by trained evaluators using tools such as the 
David P. Weikart Center's Youth Program Quality Assessment (YPQA). Using these tools, 
evaluators may observe factors such as site safety, standards of behavior for youth 
and adults, youth engagement, and adherence to other best practices. Use in M/S: 
Although these program quality assessments do not directly measure youth outcomes, 
many Ml systems can store the evaluation results of each program and compare these 
scores to the achievements of that program's youth participants (measured separately). 

Parent, student, and teacher satisfaction is often measured by survey, and sites 
funded by federal programs such as 21 s * Century Community Learning Centers are 
required to gather this information. Use in M/5: Many Ml systems can generate online 

( Continued ) 
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(Continued from previous page) 

surveys of students and distribute these surveys by email to parents and teachers. 
Responses are automatically recorded and associated with the individual student 
and program. 

Outcomes of youth who attend afterschool programs are often measured 
against the outcomes of youth who do not receive this same support. Different programs 
will be interested in different youth outcomes, but outcome measures tend to be of two 
main types: 

• Academic outcomes, including ranks, report cards, grade completion, 
achievement on standardized tests, behavior and - more generally - college 
and career readiness. Use in MIS: Ml systems record afterschool attendance, 
participation and services received by individual students. By linking this "dos- 
age" information against school and other information systems, programs can 
assess their impact on youth outcomes outside of their direct control or measure- 
ment. 

• Youth development outcomes, such as social and emotional well-being, 
community engagement, and healthy behaviors as assessed by tools such the 
Search Institute's Developmental Asset Profile (DAP). Use in MjS: Many of these 
assessments can be administered using an Ml system's web interface, and 
automatically scored, stored, and compared across programs and over time. 

As Meridith Polin, program director for Public/Private Ventures, notes, it is important to 
realize that these kinds of outcomes must be measured carefully and may take years to 
materialize. There is often a "stepped" process of implementation, says Polin, where 
agencies begin by tracking attendance, use this information to inform and improve 
their practice, reflect on the underlying factors that influence the success of youth in 
their programs, and - bearing all of this in mind - begin measuring and comparing 


4. FACILITATE PEER BENCHMARKING AND PROFESSIONAL DEVELOPMENT 


“You can only manage what you measure” is an adage that succinctly describes the importance 
of information to complex undertakings like building citywide afterschool systems. In many 
cities, that information flows only one way: from providers upward, through mandatory 
reports to funders with competing requirements and little or no cross-communication. 
In this environment, even programs that are data rich are likely to be knowledge poor - 
unaware of how their progress compares with peers, unable to share promising practices, and 
unsure of how their work aligns with the community’s larger goal of ensuring that youth who 
participate in their programs enter adulthood college and career-ready. 

More recently, cities have begun to use the implementation of afterschool MI systems as an 
opportunity to begin providing sites with meaningful feedback on their success, often as part 
of broader professional development initiatives. The Children’s Commission of Jacksonville, 
Fla., for example, publishes an annual report that lists the results of every program they 
fund - by category - according to several established benchmarks, including promotion 
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rate, school attendance, and Florida Comprehensive Assessment Test (FCAT) scores. Not 
only does this provide a very public level of accountability, but the commission uses this 
information to evaluate whether their professional development efforts are successful. In 
several cases, failure to meet specified benchmarks has led to repeated visits to providers from 
the commission and, eventually, to better performance from these lagging providers. 

Many city leaders find that, in order to establish baseline measures and negotiate the conditions 
of program success with their providers, more elemental assumptions need to be defined, 
such as “what counts as attendance?” The benefits of regular afterschool attendance have 
been well documented. In practice, however, different programs may count youth differently 
— counting a drop-in and mentoring session equally, for example, or failing to distinguish 
between recreational and enrichment activities. By developing common measures for how 
sites track their work and gauge their impact and by serving as a clearinghouse for this 
summary information across many agencies and funding sources, city leaders can transform 
these streams of data into knowledge that program managers can use to identify - and learn 
from - high performers. 



SPROCKETS BRINGS KNOWLEDGE TO 
SAINT PAUL NEIGHBORHOODS 


As Harvard University's public management 
guru Dr. Robert Behn wrote, "Despite the 
universal appeal of the seductive cliche, 
the data never speak for themselves... The 
data acquire meaning only when they are 
connected to some version of reality." That 
reality, recognized Catherine Penkert, project lead for the Sprockets database and a 
youth development specialist with the Saint Paul Parks and Recreation Department, 
can vary an awful lot between neighborhoods. 

In 201 1, the Amherst H. Wilder Foundation created a Ml system to serve both St. 
Paul's Promise Neighborhood and Sprockets. 1 1 agencies are part of the afterschool 
pilot, with more being added in 2012. As the city moves toward better access to 
common afterschool measures, Penkert sees a need to provide more specific assistance 
to program sites participating in the data system and to the afterschool community 
overall. 

Now, equipped with afterschool and academic performance data. Sprockets and 
Wilder staff will work directly with local program managers to interpret and use the 
information. In addition to discussing data at an organizational level. Sprockets will 
use their "Neighborhood Network Teams" to engage a broader afterschool community 
in using this new resource. In each case, says Penkert, the question is "'OK, Wilder 
gave us this information; what does it mean? And what might you do differently, 
knowing it?' We really hope that Sprockets can play this role, helping to connect the 
research data that we're generating to actual practice." 
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SECTION 2: 


MIS READINESS 
CHECKLIST 


GETTING STARTED 


A management information system is not, primarily, a technology project. It is a process 
of aligning the goals and resources of a great number of afterschool stakeholders, 
where technology plays an important supportive role. The first rule of management 
information systems, then, is not to begin any discussion by talking about management 
information systems. 


Common Vision 

Instead, many city and nonprofit leaders begin by convening people who care about youth 
to discuss how afterschool programs could be expanded and aligned to support other 
community objectives, such as reducing youth violence and improving college and career 
readiness. Cities that can articulate clear goals for expanding their afterschool systems are 
more successful at building the infrastructure that can accomplish that expansion, though 
the “right” alignment of these goals may vary substantially between cities and is likely to 
evolve over time. 


Right People at the Table 

Broadening this initial interest group to add seats around the table for all of the other key 
community players who have the resources, concern, and clout to pursue that vision is often a 
crucial second step. Rather than describing the purpose of a “comprehensive data system,” these 
conversations present an opportunity to discuss priorities and explore how better information 
could help providers solve problems or help policymakers answer key questions. They also offer 
city leaders a chance to expand the resources available to them and form the “policy” group 
that develops a vision for afterschool investment, establishes common standards, negotiates 
information sharing relationships, and oversees the implementation of the work. 


Coordinating Entity 

A third step for this systems-building effort is the creation or designation of an executive 
body or coordinating entity to manage it and a series of ancillary decisions about how to 
govern and fund the effort. Though this entity is often a government office or a nonprofit 
organization, some cities have developed more decentralized and collaborative approaches to 
staffing their afterschool work. 
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CITY EXAMPLE: 


SPROCKETS, SAINT PAUL'S 
VIRTUAL COORDINATING ENTITY 

The Second ShiftCommission, created by 
Saint Paul, Minn., Mayor Chris Coleman 
after his election in 2006, developed 
Sprockets, an organization that operates 
as a distributed network rather than 
being housed in a single government 
office or nonprofit organization. The 
Sprockets director is based within the 
city's parks and recreation and libraries 
departments, but the network is fiscally sponsored by The Friends of the Saint Paul 
Public Library. The associate director for quality and a network organizer are located 
at Augsburg College's Center for Democracy and Citizenship. The other network 
organizer is located within the YWCA Saint Paul. 

Three committees, an operations team, leadership group and community advisory 
council, along with three neighborhood network teams of out-of-school time providers, 
structure the work of Sprockets, with additional subcommittees convened to focus on 
issues like professional development. The Saint Paul Youth Commission engages with 
Sprockets on a project by project basis and meets with the leadership group and 
community advisory council twice each year to ensure that youth voice is represented 
in larger policy decisions. The Wilder Foundation administers the network's data 
system and provides extensive technical assistance on research and evaluation. In 
all, more than 100 stakeholders participate in setting policy, engaging partners, and 
developing Sprockets' strategies within the community. 



NLCs Institute for Youth, Education and Families has developed a series of resources - 
available at http://www.nlc.org/iyef - that explore each of these steps in more depth. In 
particular, NLCs strategy guide on Strengthening Partnerships and Building Public Will for 
Out-of- School Time Programs highlights cities that have engaged a broad range of local partners, 
kept afterschool on the public agenda, and led collaborative efforts to establish a common set 
of outcomes and a shared vision for afterschool. NLC s research report, Municipal Leadership 
for Afterschool: Citywide Approaches Spreading Across the Country , includes profiles of 27 
highly advanced citywide afterschool systems, including descriptions of the coordinating 
entities that support these system building efforts. 

City leaders who are just getting started in their efforts to enhance local afterschool programs 
can refer to NLCs Action Kit for Municipal Leaders on Expanding Afterschool Opportunities. 
The City Plaform for Children and Families offers city leaders a broader framework for taking 
action on behalf of the children, youth and families in their communities. Finally, NLCs 
report on the State of City Leadership for Children and Families explores innovations and 
trends in city leadership for afterschool and other topic areas. 
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KEY AFTERSCHOOL PARTNERS AND RESOURCES 


More than one city has compared the process of building its afterschool partnership 
to making "stone soup," in reference to the well-known fable. The analogy is apt: 
citywide afterschool initiatives need policy experts, technologists, analysts, trainers, 
and champions. Rarely does a single organization have the skills broad enough (or 
pockets deep enough) to provide all of these elements. Instead, successful initiatives 
recruit help from an array of organizations that share their mission and vision: 


STAKEHOLDER 

REPRESENTATIVES 

City 

Mayor's offices, city councilmembers, city managers and the 
heads of agencies such as parks and recreation, libraries, human 
services, police, any office of youth or education if one exists and 
the workforce investment board. City leaders can leverage and 
align valuable resources in support of afterschool programs. 

Schools 

Superintendent's offices and any relevant institutional research 
staff from the public schools, especially if many of the afterschool 
programs in the community are school-based or if academic 
information is important to the evaluation of afterschool programs. 
Nearby colleges and universities can also serve as a resource for 
research and evaluation. 

Funders 

United Ways, local and national foundations, and local philanthro- 
pists. Funders are often allies in setting quality and data standards 
for providers not directly funded by the city. They may also take 
a more direct managerial role in developing and piloting an Ml 
system, as in Grand Rapids, Mich., and Winston-Salem, N.C. 

Statewide 

Afterschool 

Networks 

The 39 statewide afterschool networks funded by the Charles 
Stewart Mott Foundation. These networks foster powerful statewide 
partnerships to impact state policy on afterschool, increase funding 
and improve the quality of programs. They are frequent advisors 
to citywide systems-building efforts. 

Nonprofits 

Major providers such as the YMCA and Boys & Girls Clubs. 
These providers are likely to have Ml systems that need to be 
accommodated, and may contribute their expertise. With smaller 
budgets and more limited capacity, smaller nonprofits have a 
separate set of needs. 

Community 

The faith and business communities. Faith-based organizations 
provide afterschool care in nearly every city surveyed by NLC, 
while chambers of commerce representatives and other business 
leaders play an important role in many citywide systems. 


Any number of commercial or self-built MI technologies can be the right fit for a city’s 
afterschool system. Some cities, such as Denver, Colo., and Boston, Mass., are designing 
citywide systems to accommodate afterschool providers who have already invested in 
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applications to track attendance and outcomes at the program level. Others, such as 
Fort Worth, Texas, were able to start nearly from scratch and provide a single system 
to their school-based providers. Several cities, such as Omaha, Neb., and Philadelphia, 
Pa., are interested in using MI systems and data warehouses to evaluate their afterschool 
investments against a broad array of education, health and human services data. Others, 
such as Providence, R.I., have tended to emphasize direct youth development outcomes, 
attendance, and surveys of students and parents. These very different business cases are 
supported by differently-configured MI systems. 

The vendors described in Section Four of this report can accommodate all of these 
strategies to varying degrees, and each has received very favorable reviews from municipal 
clients interviewed by NLC. It is equally true, however, that without a deliberate and 
inclusive process for selecting and implementing a system, these same technologies will 
disappoint. 

Successful cities have generally taken each of the following steps: 

1 . Conduct a self-inventory. City leaders may first ask what information systems 
and reporting relationships already exist. It is increasingly rare that any city is 
in a position to create a management information system from scratch. Many 
afterschool programs already use one or more reporting systems and are com- 
mitted to certain technologies. Large and small nonprofits, community-based 
organizations and faith-based providers may have very different constraints and 
concerns even within the same city, and these are important inputs into the 
decision-making process. In fact, many citywide systems develop through a 
process of evolution rather than revolution. 

2. Develop shared measures and outcomes. The information afterschool 
programs collect to manage their programs and meet reporting requirements is 
often just as diverse as their operational constraints and information systems. By 
creating a “data dictionary,” establishing common benchmarks, and harmonizing 
reporting requirements, city leaders create more efficient afterschool systems 
that are better aligned with citywide strategies for youth development. 

3. Describe the high-level business requirements. City leaders rarely complain 
that their specification for a major technology purchase was too detailed. It 
is not uncommon, however, for cities to realize late into the procurement 
process that they require user roles, case management functionality, or grant 
management features that they had not initially envisioned or for which they 
had not contracted. A description of how each of the participants in a citys 
afterschool network - administrators, agencies, providers and evaluators - will 
need to use it provides city leaders with both the criteria for selecting a vendor 
and a system specification to guide that company’s work. 

4. Design the network and establish information sharing agreements. An MIS is 

built from several key elements that may be located within different organizations 
from city to city, depending on the afterschool partnership and the resources 
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of each member. Decisions about where to host, link and aggregate data, what role 
schools or external consultants play in evaluation, and who administers the software 
can lead cities to design very different “network architectures.” These decisions also 
raise legal and privacy concerns, which must be negotiated in advance (see Section 
Three of this report on Data Stewardship). 

5. Pilot the system. To troubleshoot the inevitable glitches, build trust, and win advocates, 
city coordinating entities often opt to pilot MI systems with a limited number of their 
most enthusiastic providers. 

6. Expand and regularize. Many of the challenges uncovered during the pilot stage, such 
as an ongoing need for training, inconsistent definition of basic terms like “attendance,” 
and varying levels of data quality among providers, lead cities to explore different strategies 
for expanding their network and creating mechanisms for continuous improvement. 

Cities interviewed by NLC have been able to move through these steps at different speeds, 
but sourcing and implementing a management information system takes time. The need to 
engage and consult partners within city government, schools and throughout the community, 
in particular, requires patience. As the director of student programs in the Denver Mayors 
Office of Education and Children, Maxine Quintana, said: 

“Were not going to build this data system and then ask people to participate in it. 
Community-based organizations representing a wide variety of afterschool programs 
have been part of the system-building conversation from the beginning to help build 
this system. Their participation in the decision-making process is a huge benefit. What I 
will say is that an inclusive engagement process takes a ton of time. ” 


SELF-INVENTORY 


Dozens of nonprofit and community-based programs, schools, city recreation departments, 
libraries, faith-based organizations and others provide afterschool services in most cities. 
City coordinating entities oversee substantially less than half of these organizations, and do 
not necessarily know how the other programs are tracking and using information on their 
participants. 

A first step toward implementing a citywide MI system, then, is to find out as much as 
possible about existing programs. In some cases, cities undertake this self-inventory as part of 
a general survey of afterschool program capacity. For example, in 2006, the Omaha mayor’s 
office contracted with the University of Nebraska to prepare a Youth Afterschool Needs 
Assessment (available at www.nlc.org/afterschoolmis) that guided the creation of the city’s 
Middle School Learning Center Initiative (MSCLI). The study included a survey of parents, 
an inventory of afterschool program providers, and a geospatial analysis of underserved areas. 

While a clear sense of the overall mix of existing afterschool programs and capacity of the 
local providers offers important context, developing a citywide MI system requires more 
detailed information about the collection and use of program data. Key questions include: 
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Bridge 

beast 


Bridgeport After School Network providers ore concerned that they will 
be asked to enter information into yet 'one more' data management 
system. Therefore , whatever we do develop here in Bridgeport must 
draw from current data management systems to avoid duplication. " 

~ Tammy Papa, Lighthouse Program Director City of Bridgeport, Conn. 


What Information Systems are Already in Use by Providers? 

A city’s larger providers such as Boys & Girls Clubs and the YMCA may already have purchased 
a MI system or developed their own system. The variety of existing systems across the city 
will have to be accommodated by any new citywide system, with an aim to consolidate rather 
than multiply the number of information systems that afterschool providers have to manage. 
In addition, identifying local experts in one or more of the leading MIS products can be a 
great asset to cities. 

What Compliance Reporting Relationships and Needs 
Exist at each Program? 

Providers describe having to use as many as five - and in one case nine - information systems 
to fulfill all of the reporting and compliance requirements associated with operating their 
programs. These systems may include grant management, finance and MI applications 
preferred by each of their funders, including other city departments. “Compliance fatigue” 
can be a major obstacle to implementing any new citywide system. City leaders may look for 
ways to reduce this by consolidating systems, harmonizing reporting requirements among 
funders, or assuming some of the responsibility for compiling and distributing these reports. 


With Whom do Providers Share Information? 

Access to academic information is particularly important for programs receiving 21 st Century 
Community Learning Center funds. Sites may receive information directly from teachers, 
principals or parents, through their evaluators, or as part of a more formal data sharing 
agreement with the local school district(s). Some sites collect written permission from parents 
to share their children’s information with partners for a limited set of purposes, others do 
not, and not all programs understand or follow best practices in this regard. Standardizing 
these permissions processes and facilitating the exchange of information among afterschool 
providers and schools is a prominent way for a new citywide MI system to add value. 


How are Providers Using Information? 

Are They Interested in a More Integrated Data System? 

Most programs have specific procedures for using information that - while not always 
sophisticated - are integral to the way they manage sites, evaluate their performance, and 
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comply with requests from partners, sponsors and funders. Negotiating how to support, 


are not just going to give something away without expecting something in return.” The after- 


which organizations have the capacity, expertise, and enthusiasm to help build that new MIS, 


natt, coordinator for the City of Louisville Office of Youth Development, says, “Its always 


Do Sites Have the Skill and Equipment to Implement an Ml System? 

Program staff may need assistance overcoming specific challenges related to new information 
systems. Some sites lack computers and Internet connections. Employees’ computers skills 
may vary widely between programs, and typically high rates of staff turnover can make 
training an ongoing challenge. 


Are the Majority of Students Served by Providers Located 
within Schools or by Community-Based Organizations 
Without Strong School Affiliations? 

In some communities, public school districts are strong sponsors of afterschool systems- 


examples, see the Negotiating the Network Architecture section on page 29 and the 
description of the Nashville After Zone Alliance on page 35.) In others, networks of public 
agencies and community-based organizations have found it desirable or necessary to establish 
these resources independent of the schools. 


modify or abandon each of these procedures is a key part of expanding the citywide system 
and determining where professional development might be appreciated. 


In addition, bringing providers on board involves addressing the “whats in it for me” factor, 
as Laura Hansen of Metro Nashville Public Schools put it, “for every single partner. People 


school self-inventory is an opportunity for the coordinating entity to assess where providers 
are frustrated or limited and to use these “pain points” to develop a sense of how an MIS can 
meet the operational needs of youth-serving organizations. It is also an opportunity to identify 


and which are likely to need extra persuasion and professional assistance. As Rebecca Dejar- 


a mixed bag. There are always some agencies that are saying, c Yeah, that’s what we need to 
do - I’m on board, I want to be in the pilot.’ And then there are others that raise their hands 
and ask, c How much is this going to cost us? What’s the return on my investment for this?”’ 


building, providing leadership, technology, and research and evaluation resources. (For 






FROM DIVERSE ACTIVITIES, 


COMMON OUTCOMES IN BOSTON 

Chris Smith moved from the Boston Private 
Industry Council to run Boston After School 
& Beyond (Boston Beyond) in late 2008. 
As executive director, he was interested 
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in better aligning Boston Beyond's work with the city, funders, and the schools, 
increasing his organization's support of the city's approximately 700 afterschool 
providers, and focusing providers' attention on measurable youth development 
results. One of the chief problems, said Smith, was that "there was no coherent data 
collection and sharing strategy." 

The results of a survey his office issued in 2010 described several of the challenges 
in greater detail. While a few programs used the Boston Public Schools-based MIS, 
most providers used Microsoft Excel or had no electronic system at all. Few programs 
had access to information from the Boston Public Schools or collected permission 
from parents to obtain that information. Those providers that gained access to student 
information had mostly done so by developing personal relationships with individual 
principals and schools. 

There was, however, great interest from providers in being part of a Boston Beyond- 
led process to solve these problems and contribute to a common outcomes database. 
"We realized," said Smith, "that what we really wanted to do was address the 
barriers to a common data collection and sharing system." The additional attention 
afterschool programs began to receive as a priority of the mayor, the superintendent 
of the Boston Public Schools, and funders underscored this need for a citywide 
data system. 

To begin, Boston Beyond created a data collection pilot with five of the city's biggest 
and most enthusiastic providers. "Even among these five organizations," said Smith, 
"we saw very different approaches to data collection and saw different agreements with 
schools guiding how they got data." Together, they developed a set of data standards 
- a limited data dictionary - and a new set of expectations around sharing information. 

To extend participation to a larger cohort of the 700 providers, Boston Beyond is ex- 
ploring a "federated" system that uses an online database of providers co-managed by 
Boston Beyond and the City of Boston as a portal through which providers can upload 
their program results in spreadsheet form. This model may be adapted even further to 
serve the city's Promise Neighborhood and a Wallace Foundation-funded arts expan- 
sion initiative. Boston is succeeding in using this citywide tool to serve a number of pur- 
poses without trying to be the primary enrollment system for each afterschool provider. 

The result, says, Smith, is that "providers can be more intentional" and speak directly 
to outcomes rather than processes: 

If we can identify and measure the factors that really influence student success, 
then your typical middle school provider doesn't say "I run an afterschool 
program and we do arts - drawing on Tuesdays and painting on Thursdays" 

- and then finally get to the punch line and say "and through that we build 
teamwork skills." I want to create a discussion where they say "I 
am an afterschool provider and we build communication and 
teamwork skills through the arts, and here's how we do it. We 
lead with the skills that we want to develop." 
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DEVELOP SHARED MEASURES AND OUTCOMES 


The self-inventory described above often reveals an incredible diversity of practices around 
tracking, recording and using information among afterschool providers in each city. This 
diversity reflects underlying differences in the type and quality of programming across sites: a 
mentoring program may think very differently about attendance than a drop-in recreational 
program, for example. Those two programs are also likely to find very different types of 
demographic and activity information useful to their evaluation and management routines, 
and therefore to track relatively few items in common. Exacerbating these differences may 
be additional variation in reporting requirements from multiple funders, each of which has 
slightly different rules for how the programs they support ought to measure, aggregate, and 
segment this information. 

In order to take advantage of their investment in an MI system to develop common approaches 
to improving program quality, measure outcomes across sites, and identify strategies that are 
effective and cost-efficient, cities generally find that they and their many local providers need 
to agree on a common language and common measures for speaking about and evaluating 
afterschool activities. 

Create a "Dictionary" of Data Definitions 

A key aspect of establishing common measures is for all afterschool system participants to 
develop shared definitions for important activities and outputs. It is crucial for program 
evaluation, for example, that a city’s programs have developed a syntax in which they are not 
only using the same language, but using the same language to describe the same phenomena. 
Through an iterative process, cities tend to discover inconsistencies in how different members 
of the network record their progress and, over time, align outcome measures so that they are 
able to speak more confidently about the impact of specific interventions. 

As part of this process, some cities also develop a “taxonomy” to describe the types of 
afterschool programs they fund and support. Organizing the multitude of local programs 
into specific categories allows policymakers to quickly assess how many youth are attending 
arts-based afterschool programs compared with academic enrichment programs and to 
distinguish between outcomes in school-based and community drop-in programs. 


Establish Common Measures 

It is difficult to have a citywide conversation about improving program quality if providers 
have not agreed to a uniform set of quality standards. Establishing those standards requires a 
consensus-building process around what cross-program indicators would look like, as well as 
how they would respect and accommodate agencies’ particular needs. 

The experience of the Hartford Connects initiative is common, as described by Rachel Botts, 
the city’s former program performance manager: 

In the beginning, we just wanted everyone to buy into this so much that we said yes to 
everybody and to everything. We caused ourselves a headache, quite honestly. Because 
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this agency wanted this tweak, and that agency wanted that tweak, and this agency 
wanted three tweaks and a bucket. And the next thing you know, we were managing an 


more on the impressive progress of the Hartford Connects initiative, see page 42). 

“Data runs in herds,” as Results-Based Accountability founder Mark Friedman writes, and 
many citywide systems only gradually narrow their focus to a manageable set of variables 


outcomes related to financial sustainability, program-level outcomes around staff ratios and 


Harmonize Reporting Requirements 

Ideally, local funders of afterschool programming such as city agencies, United Ways and 
community foundations will lead this process by revising their reporting requirements to 


but it can also sharply reduce the administrative burden on program operators, many of 
whom receive multiple sources of support. 




Each member of a management information system network - on-site staff, program 


“use cases” is often the bulk of the work. 


Chart 4. Reporting Capabilities and the Ability to Protect Data are the Top Two MIS Needs 


Cost 

Ease of migration 
Integration with other city / school systems 
Ability for IT staff to customize 
Turnkey Product 


enterprise where there’s not enough alignment - where we thought there would be (for 


that can serve as effective proxies for the rest. These indicators commonly include city-level 


parental involvement, and youth-level outcomes that might include sustained afterschool 
attendance, on-time grade promotion, and pro-social indicators of confidence and character. 


better correspond with one another and with the newly-developed citywide standards. Not 
only does this allow for a better comparison of program activities and results across systems, 



managers, city administrators, evaluators and others - has a specific set of requirements from 
an afterschool management information system. Exploring and documenting these uses is a 
painstaking and collaborative process, but the result is a well-defined set of specifications that 
become the basis for a city’s request for proposals and implementation plan. Defining these 


Reporting capability 


Security and privacy of data 
Customizable user interface 


Customer support 



o% 

J Very important Somewhat important 


10 % 20 % 30 % 40 % 50 % 60 % 70 % 80 % 90 % 100 % 


Source: 2011-2012 survey of city leaders by NLC 
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Each networks system specification will be different. Sometimes, the system requirements are 
driven by the priorities of the city leaders and/or coordinating entity that funds and builds 
the MIS. In other cases, the challenges and opportunities identified by the system partners 
through a self-inventory play a larger role. Ideally, these two sets of needs correspond to one 
another and are further developed through conversations with MIS vendors and other experts. 

Below is a checklist of many of the common functions that cities contract with MI system 
vendors to provide. A version of the request for information (RFI) developed by NLC to 
compare six commercial MIS vendors is available online at www.nlc.org/afterschoolmis. 
We encourage city leaders to download the model RFI and adapt it as part of their own 
procurement process. 


MIS FUNCTIONALITY CHECKLIST 


ENROLLMENT, ATTENDANCE AND PARTICIPATION 

D Enroll and register youth in programs 

□ Create printed rosters 

D Track attendance using a computer or mobile device 

□ Track attendance using a card reader or scanner 
D Track activity participation (dosage) 

D Attribute participation data to specific grant-funded programs for reporting purposes 
D Group family units (i.e., affiliate youth with parents or siblings) 

□ Option to import enrollment/attendance information from other Ml systems 

CASE MANAGEMENT AND REFERRALS 

D Record individual youth service plans and milestones 
D Record staff interactions and communication 

D Facilitate referrals across organizations that do/do not participate in the Ml system 

AGENCY, SITE AND STAFF INFORMATION 

D Record site and organization characteristics such as program quality ratings, 
program location, services, presence of bilingual staff, and transportation options 

□ Record information on staff such as credentials and years of experience 

□ Associate staff with programs and program sites 
D Publish an online provider directory 

SURVEY AND EXAM ADMINISTRATION 

D Administer surveys and assessments to youth (such as the Survey of Afterschool 
Youth Outcomes or the Search Institute's Developmental Assets Profile) 

D Administer online surveys to others (e.g., parents, teachers, or instructors) 

D Vendor has preloaded survey and youth assessment instruments 
D Enable local administrators to create custom surveys 

D Enable vendors to implement new survey instruments for the city (and include 
information on cost/terms of delivery) 

(Continued) 
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REPORTING 

□ Provide built-in compliance report templates (list any required, such as 21 s * CCLC) 

□ Pre-build stock reports for administrators, agencies, and programs 

□ Enable local administrators to create new reports 

□ Enable vendor to create new reports (and include information on cost/terms 
of delivery) 

□ Integrate with business intelligence applications (such as SAP/Crystal Reports) 

□ Offer advanced reporting features (e.g., an RBA Scorecard, dynamic reports 
with Microsoft Live, a management dashboard) 

INTEGRATION WITH OTHER DATA SYSTEMS 

□ Provide an API or another kind of interface that allows third-party applications 
to extend the functionality of the software 

D Aggregate information from third party afterschool Ml systems 

□ Link to school student record databases 

□ Provide afterschool sites with information on individual students (alternative: 
aggregate information only) 

□ Provide afterschool sites with real-time student information (alternative: periodic 
or annual) 

□ Provide sufficient user roles and suppression of private data to avoid violating FERPA 

TRAINING AND USER SUPPORT 

□ Provide training to sites, agencies, administrators (list requirements) 

□ Make training available from vendor or user group thereafter (including information 
on cost/terms of delivery) 

□ Make customer service available (including information on cost/terms and medium 
of delivery) 

This checklist is not meant to be comprehensive. More information on each of these 
functionalities is available through the vendor comparison in Section Four. Cities may 
require additional features such as grant management or longitudinal data on participants 
that are not part of a “core” afterschool MI system, or may have very specific needs around 
connecting youth activities and case management services with particular funding sources 
for billing and reporting purposes. 

Often, the process of defining business requirements and identifying the best way to meet 
them is an iterative one. Afterschool providers will express their specific needs and limitations 
around technology, training, and information for generating grant requests and reports. 
School districts and other informational partners may suggest a path for negotiating privacy 
concerns or raise objections to a city’s evaluation plans. Vendors, once they have a fairly clear 
sense of a network’s needs, can provide invaluable consultation around what strategies have 
worked in other cities. This is a “rolling process of discovery,” as one major vendor described 
it, and city leaders should not expect to single-handedly draft an RFP and issue a contract. 
Most cities should, however, consult with multiple vendors about their needs. 
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CITY EXAMPLE: 


SELECTING A VENDOR IN 
WASHINGTON, D.C. 

In 2008, the DC Children and 
Youth Investment Trust Corpora- 
tion, a nonprofit intermediary 
that plays a key afterschool 
coordinating role in the District 
of Columbia, had an incum- 
bent Ml system that they felt the 
organization had outgrown. 
Keith Watson, a former Trust 
employee and president of Kai- 
ros Management, was hired to manage the process of replacing it. He and Natasha 
Harrison, the Trust's grants management and training consultant, began by thinking 
backwards from the result they sought: 

What data are we going to need to get out of the system at the end point? If you 
imagine some future state where the system's been in use for a couple of years, what 
kind of data do we want to be able to pull out? What kind of reports? How do we 
want to manage performance? What statements do we want to be able to make to 
stakeholders about who we served and what outcomes we've achieved? 

Watson mapped out the roles of each of the Trust staff and their grantees and charted 
how they might interact with a new system to track student attendance, review 
management information and manage grant cycles. This detailed work is crucial, said 
Harrison. "You have to make it very clear to everybody - and to your vendor - what 
you need. Customizing sounds great, but the lesson we learned is to be very specific 
and be sure what you are getting will [work when it is first implemented]." 

With this system specification roughed in, Watson issued a 14-page request for infor- 
mation. A lot had changed, he said, since the Trust had selected its first system. They 
wanted to understand the state of the art, to get smarter about the market, and to get 
a better sense of what kind of money they might have to spend. 

The Trust's request for proposals, issued several months later, reflected everything 
the organization had learned. It was tighter, its demands were more specific, and 
Watson was confident in the rubric he had designed for an internal committee to 
score and rank the responses. Several vendors were invited to make presentations, 
and two were clear favorites. While an external group of grantees tested both systems 
and provided feedback, the Trust negotiated a final offer price with both companies. 
One of the lessons of this process, said Watson, was that "having multiple vendors 
competing under a formal negotiation process made a big difference." 

For more information, contact Keith Watson at Kairos Management at 
www. Ka i rosMg t. com . 
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CITY EXAMPLE: 


OMAHA BRINGS EVERYBODY TO THE TABLE 

Citywide afterschool systems-building in Omaha 
began in 2007 as a partnership between 
Mayor Mike Fahey, the Omaha Public Schools, 
the Sherwood Foundation and Building Bright 
Futures, a public-private partnership that seeks 
to improve academic performance of students in 
the Omaha metropolitan area. A thorough survey 
conducted by the University of Nebraska at Omaha in 2006 provided a foundation 
for understanding the city's existing afterschool landscape and led the partners to 
focus on providing high-quality programs as part of a new Middle School Learning 
Center Initiative (MSLCI). 

A new and relatively small organization, Collective for Youth (CFY), was created as 
an intermediary to oversee these programs, with the Omaha Community Foundation 
serving as its fiscal agent. Though CFY worked very closely with the mayor, the 
partners decided that the initiative had a better chance of being sustained over time 
if it was embedded within the community. 

Mayor Jim Suttle succeeded Mayor Fahey in 2009 and, shortly after taking office. 
Mayor Suttle issued a proclamation recognizing afterschool programming as a city 
priority. With his office's support and the support of the Omaha Public Schools, 
CFY began negotiations in 2011 to expand beyond MSLCI to manage 18 of the 
school district's 21st Century Community Learning Centers for elementary students, 
with a goal of eventually managing all of the district's afterschool youth development 
programming. In mid-2011, CFY received its nonprofit designation from the IRS. 
Concurrently, Building Bright Futures, in partnership with the Sherwood Foundation, 
Avenue Scholars, and the local United Way, started to explore options for a 
management information system that could support this expanded scope of work and 
to replace the Microsoft Excel spreadsheets that would soon be insufficient. 

Building Bright Futures led an inclusive MIS selection process with a "data collaborative" 
consisting of representatives from United Way of the Midlands, major afterschool 
providers such as the Boys & Girls Club, community-based organizations, and 
nonprofits operating mentoring programs that extend beyond traditional afterschool. 
As described by CFY's executive director, Megan Addison, the collaborative "looked 
at all of the data that we needed to collect and created a data dictionary so that all 
of us would be entering the same information consistently. For example, if we need 
to create reports for evaluation or reporting purposes, one group isn't defining race 
in a way that's incompatible with the way another group is doing it - so that was all 
figured out." 

Four MIS companies presented their products to stakeholders in early 201 1, and 
members of the collaborative ranked each of the applications individually, ultimately 
selecting the proposal from nFocus Solutions. By August 2011, several MSLCI 
afterschool sites were actively using KidTrax, the nFocus data management tool 

(Continued) 
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for tracking individual program participation. Building Bright Futures began helping 
additional partners implement the KidTrax system. An "attendance collaborative" and 
an academic support and reengagement program are currently using the Ml system's 
case management features. A mentoring group and a teen and young parent program 
will begin using KidTrax in early 2012. In addition to KidTrax, a vitally important 
aspect of Omaha's collaborative is the eventual ability to aggregate attendance and 
outcomes information from different providers using the nFocus Solutions Community 
Server platform. 

"One key to the relatively smooth rollout," says Addison, "was Omaha's good fortune 
in having a person familiar with the nFocus MIS within our community. Boys & Girls 
Clubs also used nFocus, and Building Bright Futures and CFY were able to borrow their 
system administrator for nearly a year to get each of the pilot sites up and running. 
Each site's interface was customized to accommodate their unique application, but 
with attention paid to ensuring that all sites tracked the fundamental data elements 
required by the partnership to allow comparison of outcomes across programs." 

Not all of the work has been this straightforward. Using an MIS to store student academic 
data raised new privacy concerns for the Omaha Public Schools. It required more 
than a year of negotiation to establish a new memorandum of understanding with the 
district that allows selected student records to be matched against youth development 
outcomes. Said Addison, laughing, "It was a long, drawn out process! But in the end, 
it got us the buy-in before the system was even purchased, which I think was key." 


DESIGN THE NETWORK ARCHITECTURE AND NEGOTIATE 
INFORMATION SHARING AGREEMENTS 


Cities can build management information systems in a number of ways, depending on their 
requirements and the resources available to them. Choosing a design requires answering 
several key questions (and options for addressing each question are illustrated by the city 
examples that follow): 


Who Owns the Information? 

An early and important action for city leaders, having brought the right people around the 
table, is to begin discussing the conditions under which these partners may be willing to share 
their data. As the Urban Institute s National Neighborhood Indicators Partnership describes 
in detail in their online guide to data sharing, patience, respect, organization, and knowledge 
of the relevant state and federal regulations are all crucial to a successful negotiation. 

For most cities, establishing an information sharing relationship with the public schools 
is the most difficult challenge - in many cases taking more than a year. Section Three of 
this report describes the Family Educational Rights and Privacy Act (FERPA), the federal 
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law most often cited as a barrier to sharing information between schools and afterschool 
providers, and the main components of a memorandum of understanding. Several excellent 
resources are additionally available to city leaders: 

• The Master Data Sharing Agreement (MDSA) developed between the Grand Rapids 
Public Schools, the Community Research Institute at Grand Valley State University, the 
Doug and Maria DeVos Foundation and Calvin College’s Center for Social Research is 
available by request through an online form at http://www.cridata.org/b2bmdsa/. 

• An article describing the development of Grand Rapids’ MDSA was published in 
November 2011 by the Johnson Center for Philanthropy at Grand Valley State 
University in Volume 3, Number 4 of The Foundation Review (see pp. 14-33). 

• The Urban Institute’s National Neighborhood Indicators Partnership online guide 
to data sharing is available at http://www.neighborhoodindicators.org/library/guides/ 
nnip-lessons-local-data-sharing. 


Who Will Administer the Ml System Database? 

In some cities, such as Nashville, Tenn., the afterschool enrollment and participation database 
is an extension of the school district’s student record system and is hosted by the district. 
In other cases, as in Denver, Colo., afterschool enrollment and attendance information is 
recorded in an independent database purchased and managed by a coordinating entity such 
as the United Way or a municipal office. In at least one case (Grand Rapids, Mich.), a 
university research center administers the MI system. 


Where and How is the Link between Academic Records 
and Afterschool Enrollment and Attendance Made? 

Students’ afterschool participation and academic outcomes are most often tracked in at 
least two separate databases by two or more separate institutions. To allow these multiple 
systems to communicate with one another, each student’s record must be linked against its 
corresponding record in the other database(s). Two strategies are available: 

• Common ID: If the databases share a unique identification code for each student, 
such as a student identification number, records can be easily matched and transferred 
between them. In the case of the Nashville After Zones, for example, the school district 
creates every afterschool provider’s roster, which includes a code so that each student’s 
information can be easily matched with academic outcomes stored in the district’s data 
warehouse. 

• Probabilistic matching: Often, however, MI systems must use information such as 
student first and last name, date of birth, and family address to make an educated guess 
about which records in two or more databases correspond to the same youth. This 
algorithm-driven process, which is never 100 percent accurate, is known as a making a 
“probabilistic” match. 
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Again, this match can be made in real-time within an MIS hosted by a school or coordinating 
entity, it can be made in a data warehouse, or in other cases the information may be linked 
offline by an external university evaluator who receives separate files of academic information 
from the schools and afterschool participation from the coordinating entity. 


What Organization is Responsible for Analyzing the Information? 

Very often, schools are the ideal place to analyze afterschool data, as they have much of the 
relevant data and are not constrained by FERPA when analyzing data internally. In Bridgeport, 
Conn., for example, the city’s Lighthouse program provides its afterschool participation 
information to the public schools for analysis, and the Mayor’s Office of Education and Youth 
Services receives back aggregate reports organized by a predefined list of demographic and 
program characteristics. On the other hand, says Boston Beyond’s Chris Smith, “Districts 
- and especially their research offices - tend to be overwhelmed with data requests. So that’s 
not always the best way to go.” Nor does it provide great flexibility to afterschool staff to run 
more complex analyses if they find themselves on the wrong side of the FERPA “firewall” (see 
Section Three for several possible strategies). Instead, coordinating entities may negotiate the 
resources and access to do the work themselves or designate a third party researcher. 


BUILDING TRUST WITH A QUALIFIED EVALUATOR 


Leaders in cities such as Bridgeport, Conn., Fort Worth, Texas, and Jacksonville, Fla., 
made a special effort to describe how important it is for afterschool program coordinators 
to have access to an evaluator who "really understands the work," and has evaluated 
afterschool programs over a long period of time. 

To Miguel Garcia at Fort Worth After School (FWAS), which has a longstanding 
relationship with faculty at Texas A&M University, it is valuable to have a partner that 
is scientific minded and that can be honest about the evidence - "what correlations are 
real, which are spurious" and how FWAS ought to talk about its impact. 

Jacksonville's evaluator has been working with the Jacksonville Children's Commission 
for a decade, and also works closely with the school system. Said Dolly Dillin, director 
of grants administration for the commission, "she's just an expert. Not everyone can 
understand this information." So strongly did the commission feel about its evaluator's 
expertise that it continued to rely on her even after she relocated to Boston. "She 
understands Jacksonville, and that's important because each community is just so 
different," said Dillin. 

What this comes down to, according to Bridgeport, Conn., Lighthouse Program Director 
Tammy Papa, is trust. The 201 1-12 evaluation of Lighthouse conducted by its evaluator, 
MRM, will be the company's tenth. 


City leaders must be aware that choosing a single company or software platform to serve 
several agencies and programs does not, by itself, guarantee that each of those agencies 5 
systems will be able to communicate with one another. It is not unusual for the same product 
to support multiple agencies independently within the same city or for the same product to 
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be used by both the state department of education and a local citywide system. In these cases, 
program providers may find themselves entering information more than once into two or more 
versions of the same software package for different funders. Having the city’s providers use 
the same product for managing their information is not the same as coordinating the city’s 
providers through one system. The difference is in the design of the network. 



CITY EXAMPLE: 


A FEDERATED MIS IN DENVER 

Denver's Lights on After School (LOAS) 
initiative was created in 2003 and has 
directed more than $7 million to serve 
more than 10,000 students annually. 
LOAS has also offered professional 
development to more than 100 staff each 
year. The Mayor's Office of Education and Children, the Denver Public Schools (DPS) 
Department of Extended Learning and Community Schools, the Denver Public Schools 
Foundation, and the Mile High United Way are all close partners in a new citywide 
effort, the Denver Out of School Time (OST) Alliance, which is being supported by 
a new round of investments in citywide afterschool systems-building by The Wallace 
Foundation. The technology partner to this group is CiviCore, a company formed in 
2000 with close ties to Denver's Piton Foundation and a large footprint in the city's 
nonprofit community. 

Rather than build a ground-up application to track youth afterschool enrollment and 
attendance, CiviCore and the Denver OST Alliance are focusing on two areas of 
particularly high value to the partnership: 

1 . Online program directory: Denver has no complete inventory of the city's 
afterschool providers, and CiviCore is developing a platform where community 
programs can self-populate with information on the scope of their services. This 
platform will include information about program locations, with which schools 
programs are partnering, the number of students they serve and high-level 
program outcomes - all arranged within a "taxonomy" of program types that 
include academics, arts/culture, leadership, life skills, recreation and technology. 

2. Common data tracking: Developing a shared school-afterschool database 
is crucial for Denver's partnership. Currently, DPS is overwhelmed with requests 
from community providers for academic data, and no unique ID exists to track 
youth participants between programs or over time. CiviCore is working closely 
with school officials to develop a protocol whereby afterschool programs 
are likely to use the DPS student ID for each of their participants and will use 
Civicore's Community Partnership System to share participation data and 
retrieve academic information. 

Through this "federated" model of afterschool information sharing, providers will 
continue to use the systems they prefer to track enrollment and participation - whether 

(Continued) 
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spreadsheets or other software applications. The citywide system is coordinating this 
at a high level, so that policymakers have as complete and accurate an inventory of 
afterschool programs as possible, have unduplicated counts of participation across 
the city, and can pool data to begin to draw conclusions about the scope and 
effectiveness of these services. In the process, the Community Partnership System 
is addressing two "pain points": the need for DPS to reduce the overwhelming 
number of individual requests for student information, and the need for community 
organizations to have a straightforward protocol for receiving academic outcome 
information on the youth they serve. 



THE BELIEVE 2 BECOME 
INITIATIVE IN 
GRAND RAPIDS 

The "Believe 2 Become" 
(B2B) initiative is designed to 
provide real-time management 
information to afterschool 
providers, including nightly 
data transfers from the 
Grand Rapids Public Schools 
(GRPS) into the initiative's 
afterschool Ml system. From 
a systems-design standpoint, it has several interesting features. A university partner, 
the Community Research Institute (CRI) at Grand Valley State University, hosts the Ml 
system. CRI is designated as an Agent of GRPS for the purposes of FERPA compliance, 
and as such is permitted to receive identified student information. CRI makes the link 
between student academic records and participation information sent to them through 
an nFocus KidTrax Ml system. KidTrax assigns B2B participants their own internal 
unique ID, and CRI maintains a crosswalk file to match these identifiers GRPS student 
IDs. Using this system, CRI collects the following information: 

• Management information: When parent permission has been granted, 
CRI sends individual student academic information into the Ml system, where it 
can be accessed by designated afterschool staff. 

• Evaluation information: CRI staff send the DeVos Foundation, which is 
funding the initiative, and evaluators at CRI, Calvin College, and Basis Policy 
Research complete but de-identified information for comparing the outcomes of 
afterschool participants to nonparticipants. 


(Continued) 
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To assist the schools in this partnership, the DeVos Foundation funds a full-time position 
at GPRS. Local leaders in Nashville are also considering this strategy as they expand 
the city's afterschool system beyond school-based providers. The progress of Grand 
Rapids' B2B initiative is the result of collaboration and support by a very active 
foundation, DeVos, two academic institutions as research partners, CRI as the data 
management partner, GRPS, and the company nFocus as a technology provider and 
consultant. 

More information on the development of this partnership and its ongoing gover- 
nance is provided on page 52, with a link to a journal article written by CRI and 
DeVos Foundation staff that describes the process on page 30. 
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CITY EXAMPLE: 



CONNECTING SCHOOL 
DISTRICT AND AFTER 
ZONE DATABASES IN 
NASHVILLE 


The Nashville After Zone Alliance (NAZA) is a partnership among the school district, 
city agencies and youth-serving organizations modeled on the Providence (R.l.) 
After School Alliance's (PASA) "AfterZones." The first two of six planned zones were 
launched in 2010 and 201 1, each with its own oversight agency, which contracts 
with local programs to offer a mix of academic and youth development activities for 
several hours after each school day. 

NAZA's management information system is built from three elements, two of them 
hosted by the Metro Nashville Public School District (MNPS) and the third developed 
by the Information Technology Services Department of the Metropolitan Government 
of Nashville (Metro ITS). These elements include: 

1 . The MNPS student unit record system: The school district's core student 
database is used to generate rosters for each afterschool program offered through 
an After Zone. District staff who have access to the school database carry out this 
work in collaboration with NAZA, which helps to oversee program enrollment. 

2. The NAZA database: MNPS exports these rosters nightly and sends them 
outside of the district's firewall and into NAZA's database, which is developed 
and maintained by Metro ITS. Only FERPA-compliant directory information on 
each student - student contact information, grade level, and a unique district 
student identification number - is made available to afterschool programs. 
The NAZA database has a simple, web-based interface for tracking program 
attendance and an administrative screen that allows NAZA leaders and 
coordinators to monitor the attendance taking process, manage user passwords, 
and do basic reporting on student program participation rates. 

3. MNPS data warehouse: The school district's fast-developing data 
warehouse receives these two streams of information separately - one from 
the student unit record system and one from the NAZA database - and uses 
the student identification number present in both sets of records to link the 
information for reporting and analysis. 

Nashville's Mayor, Karl Dean, and the superintendent of the Metro Nashville Public 
Schools, Jesse Register, have both committed significant resources to this systems- 
building effort. Concrete results of the partnership include: 

• The reporting and compliance burden on NAZA's afterschool 
programs has been greatly reduced. The MNPS data warehouse 
provides all of the attendance, behavioral and academic information necessary 
for most evaluation reports. Site managers no longer have to ask students for 
their report cards to get information on grades and attendance. 

(Continued) 
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• Providers receive a program "dashboard." This dashboard allows site 
managers to track their performance against key benchmarks and to identify 
participants who may need additional assistance due to school behavioral 
issues, absences, or academic challenges. 

• NAZA will be able to evaluate system-wide outcomes, comparing 
afterschool participation against a full range of performance benchmarks from 
MNPS, year-over-year for the same cohort of students. 

The district's sponsorship of this information system means that the informational "link" 
between participants at MNPS and the NAZA afterschool programs is seamless. 
Analysis and reports on system-wide outcomes can be produced by the data warehouse 
and the district's Research Assessment and Evaluation Department without fear of 
violating FERPA restrictions. 

Conversely, the district's role in managing afterschool program rosters from the 
student information system imposes a possible limit on the scalability of the Ml system 
to include afterschool providers in Nashville who are not affiliated with NAZA. 
According to Hansen, "The real limit is setting up the program rosters in the school's 
student management system, because that is behind the firewall - so program 
providers can't create those rosters directly. They have to have a school liaison." 
One possible solution being investigated is to use a common external data system to 
manage their programs and link that single system to the MNPS data warehouse. This 
method will be piloted in the Nashville Promise Neighborhood, which will use Social 
Solutions' Efforts to Outcomes platform as the service provider system. 



M 


OWNED BY METRO GOVERNMENT 


Roster Information 


Role Based 
Security 


© 


Section 2: MIS Readiness Checklist 


36 



BUILD OR BUY? 


Approximately one-third of the cities surveyed by NLC in 2011 elected to build their 
own afterschool Ml systems rather than purchase a commercial product. Cost was a 
motivating factor in many of these decisions. As Laura Hansen, who assisted with the 
creation of the Ml system for Nashville's "After Zones," describes her decision-making 
process: 

We really wanted to maximize the funds that were going toward programming (as 
opposed to administrative costs) and we didn't have large sums of money to do 
this. It fit with the approach of the Mayor's Office of Children and Youth to look 
at resources we already had available to use, and encourage collaboration. Both 
the city and the district technology departments agreed that there was value in 
working together and using our existing technology and talent rather than investing 
in another external information system." 

In other cases, the availability of local information technology resources encouraged 
city leaders to keep the work in-house. Miguel Garcia, the director of Fort Worth After 
School, had already developed a preliminary afterschool tracking application when 
he recognized an opportunity to partner with the Fort Worth Independent School 
District to create something much more sophisticated: 

When we talked to our technology department, they said, "Look - we've got the 
man hours. If you've got the money in your budget, we have individuals that are 
capable of creating a reliable, multi-faceted data gathering system." It's not a 
purchase order to an outside organization. 

The school district's new chief of technology had recently bought new servers, brought 
on a couple of new programmers, and was willing to support the application "for 
as long as Fort Worth After School is around." The application they developed 
together now serves as one of the templates for the system utilized by the 2 1 st Century 
Community Learning Center program at the Texas Department of Education. 

There are, however, risks associated with this approach: 

• Self-developed systems may be less flexible than commercial 
products and therefore more difficult to bring to scale. Cities that opt to build 
an Ml system as an extension to a school district's student record database, for 
example, may find themselves very closely tied to that district's students and pro- 
grams. It may be difficult to provide access to programs serving students in other 
districts or to community-based organizations that lack a formal relationship to 
the district. Local agencies that opt to build an afterschool Ml system as an exten- 
sion of a much bigger product - such as a city's Homeless Management Informa- 
tion System or social services platform - may find that they have little influence 
over the product's development path, and that updates regularly interfere with its 
usefulness to afterschool providers. 

• The Ml system may be "orphaned" by its developer(s). Several cities 
have found that product features developed by city staff or local volunteers, at 
some point, ceased to function. The online program directory crashed or the 

( Continued ) 
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software bridge that linked afterschool program data to school records broke. In 
such cases, the programmer who wrote (but did not document) the application 
has frequently moved on. In several cases, cities endured prolonged down time 
while they decided whether to rewrite the application from scratch or contract for 
services with a commercial Ml system vendor. 

• Unique Ml systems evolve slowly. In most cases, the relatively large 
installation base of commercial Ml system providers allows them to benefit 
from these economies of scale to innovate more quickly and provide more 
comprehensive customer support. Larger user groups mean that solutions to many 
challenges - such as the integration of a new youth assessment or a niche need 
like biometric identification - have already been developed and can be more 
easily adapted for local use. 

Cities that have developed Ml systems that successfully meet the needs of their 
afterschool coordinating entities and their networks' providers usually exhibit most, if 
not all, of the following characteristics: 

• Strong internal IT capacity: This capacity may be provided by the city, the 
school district, or a third entity, but having IT capacity within the partnership - 
as opposed to available on contract - is crucial to both making smart design 
decisions and then supporting the application over time. 

• Dedicated funding: Ml systems require ongoing developmentand customization 
to accommodate new needs and new partners. Dedicated funding allows the 
partnership to plan ahead for these needs and to adapt. For example, the 
Jacksonville Children's Commission is one of seven Children's Services Councils 
in Florida that partnered with each other to build their own MIS, the Services and 
Activities Management Information System (SAMIS), in 1999. Between 2005 
and 2008, these councils used SAMIS to allocate more than $1 billion in funding 
to afterschool programs and other social services. Each council contributes an 
annual fee to the maintenance and development of the system, which amounted 
to approximately $125,000 in 201 1 . 

• Close partnership between the city and/or its intermediary and 
the school district: Most self-built afterschool Ml systems leverage schools' 
investment and expertise in student data systems. Nashville city leaders developed 
this relationship incrementally, including the school district as a partner from the 
very beginning of the city's exploration of an afterschool Ml system. According 
to Laura Hansen, "Having someone on the outside, or an intermediary, do that 
visioning independently and then arriving at the schools to say 'all we need you 
to do is give us this' is not a great way to do it, in my experience." 

• Clear system specification: Even if self-built Ml systems are not as flexible as 
their commercial alternatives, they can be tailored effectively to the needs and 
technologies present in the community. Where the coordinating entity has a very 
clear sense of the network's needs and can define these needs without the benefit 
of consultation with a commercial provider, they are often successful. 
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PILOT 


Many cities opt to pilot their new MI system before deploying it more widely. They may 
reach out to volunteers identified in the “self-inventory” phase of work, and screen for 
organizations that seem to have the capacity and enthusiasm to be active thought partners. 
For example, Saint Pauls Sprockets network - which provides no funding - solicited interest 
from the city’s nonprofit providers in participating in a pilot of the city’s new Cityspan data 
system. The network selected only 1 1 of the 16 organizations that volunteered. 

This phase of work is important. Major features of the software, such as its integration with 
other systems and the usefulness of its reports, will need to be customized and trouble-shot. 
The overriding goal here is simplicity, as Juan Ruiz, the SAMIS senior administrator in 
Jacksonville, explains: 

All the time we are spending on administration is time we are not spending with the kids. 
Our programs do not have money to hire a bunch of additional administrators. So our 
job (as system designers) is to make this as simple as possible. 

Jacksonville went as far as to create a stripped-down, simplified desktop application outside of 
the normal SAMIS interface for some of its providers to enter attendance information. To get 
accurate data from staff, ease of use was more important than total uniformity of the interface. 

Boston After School & Beyond’s pilot included just five of the city’s largest providers, all of 
which were interested in learning how to contribute to and receive the benefits from a shared 
afterschool data system. Over 18 months, they identified and fixed problems with individual 
programs not collecting student IDs, developed an initial set of common data definitions, 
and settled on an information architecture that looks more like Denver’s “federated” MI 
system than a soup-to-nuts unified enrollment and tracking system. The pilot established a 
foundation for expanding the system and inviting a greater number of Boston’s estimated 
700 afterschool providers to participate. 

Finally, a pilot phase allows cities to estimate more accurately the amount of initial and 
ongoing training the MIS is likely to require. The amount and type of training needed varies 
across communities. One city provides training every week, year-round. Other cities have 
implemented similar “train the trainer” models and negotiated with commercial MI system 
providers to take advantage of periodic online webinars and customer support. Even so, 
several cities noted that there is a limit to what managers can expect of some afterschool 
organizations, with particular frustration expressed about underequipped and skeptical city 
agencies. Overcoming this resistance is easier if the coordinating entity is not stretched by 
commitments to too many sites. 


EXPAND AND REGULARIZE 


Even a successful pilot will tend to multiply, rather than diminish, the number of questions 
facing MI system managers from providers and policymakers. Success breeds more enthusiasm 
to do more with the available resources. It also tends to bring to the foreground several 
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challenges that have been outlined in this report already, including the ongoing, iterative 
process of identifying common measures and updating the “dictionary” of data definitions. 

In addition, city leaders often exit the pilot with a mandate to “think big” about improving 
their program evaluation and integrating a more diverse set of youth support programs into 
the system. 

Improve Reporting and Evaluation 

Ideally, an MI system is flexible enough to create management reports that are responsive to 
provider needs and the questions of policymakers. Structuring queries and running reports 
is an ongoing process of inquiry. 

All commercial MI systems offer “canned” reports that will meet a variety of city needs, 
and most companies include in their initial consultation an opportunity to customize and 
create new reports that are better targeted to individual needs. Over time, these needs tend 
to expand, and software packages vary in their ability to provide administrators with flexible 
“report building” tools to meet these demands. (See Section Four for a comparison of 
commercial MI systems.) 

Ultimately, however, most sites develop a limited set of reports and dashboard utilities that are 
aligned with common benchmarks such as program attendance and certain youth outcomes 
and that become the touchstones of ongoing performance management. Though these may 
change modestly as citywide priorities for afterschool are realigned, the standard for good 
reporting remains relevance, concision, and timeliness. 

Formal program evaluation, on the other hand, is a more complex undertaking. Once an MI 
system is online, providers are trained, and information is being shared between afterschool 
programs and other youth-serving agencies, data will accumulate at an astonishing rate. The 
distinction between “data” and “knowledge” becomes crucial here, as several city leaders 
interviewed by NLC admitted they had five or even 10 years of longitudinal data that 
were parked on a server, unanalyzed. “We re sitting on a ton of information,” said one city 
representative, “and we just don’t have the dollars to do anything with it.” 

This state of affairs is more the rule than the exception. While we know from longitudinal 
studies of programs like After School Matters in Chicago that high-quality afterschool is 
effective, relatively few local studies can demonstrate this impact with any rigor. Ideally, city 
leaders would prefer to be able to report to their funders, their mayor, and their public that 
their afterschool system is effective. The Public/Private Ventures report on Providence’s PASA 
initiative is an excellent recent example of such a report. 

Several cities are creating the capacity to conduct at least limited evaluation studies within 
their afterschool coordinating entity. Boston After School & Beyond recently hired a data 
manager, for example, and Chicago Allies for Youth Success has a senior systems analyst. 
More often, cities find external partners: 

• The Providence After School Alliance is negotiating with the city’s major data intermediary, 
the Providence Plan, to link afterschool outcomes with the K-12, postsecondary and 
health data already hosted by the intermediary’s integrated data system. 
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• Fort Worth uses a Texas A&M University-based evaluator with a decade-long 
relationship with Fort Worth After School to write their annual outcomes report. 

• Bridgeport, Conn., contracts with a private evaluator, MRM, Inc., to track the academic 
progress and social and emotional well-being of students who participate in afterschool 
programs. Bridgeport associates afterschool participation data with reductions in 
crime rates and examines reading, writing, and math scores by ethnicity in an annual 
evaluation of afterschool program impact. 

• Jacksonville’s private evaluator provides simple, consistent outcome measures for a host 
of local programs including “TEAM UP” afterschool and special needs programs, case 
management, mental health and mentoring programs, and other social services (all 
tracked in the city’s MI system, SAMIS). 

Cities are exceptionally careful about making extravagant claims about the value of afterschool 
programs. As Miguel Garcia, director of Fort Worth After School, reported: 

Of course being down here, you have a lot of people who will say the reason for a kid’s 
success is four-fold: school, family, community programs, and church. . .When we look 
at the data we think, “Man we’ve got to be careful about how we lay this argument out.” 
A lot of these [successful kids] had good parental support. It can be hard to control for 
all of the factors that we know, intuitively, that we ought to. 

This concern is well-founded: A decrease in neighborhood crime that occurs simultaneously 
with the expansion of local afterschool programs may be suggestive, but it is not close to a 
causal relationship. What MI systems allow cities to do is track individual student activities, 
characteristics, and outcomes. To a degree much greater than has been taken advantage of, 
to date, this enhanced data capacity will allow skilled evaluators to control for many of the 
environmental factors mentioned by Mr. Garcia and to make a much stronger claim for 
afterschool’s value to youth. 


Integrate More Data 

A growing number of cities are beginning to merge information from multiple agencies and 
service systems outside of education and youth services, and to use the insights gained from 
the analysis of this linked information to create more effective program interventions, make 
smarter policy, and improve the care provided by educators and social workers. 

These so-called “integrated data systems” directly support the mandate for public managers 
to provide more coordinated care by including information from health and human services 
departments, workforce investment boards, and juvenile courts when making decisions 
about young people. 

• In Philadelphia, a majority of city-funded afterschool providers use a self-built MIS, 
the Provider Contract Attendance and Performance System (PCAPS), operated by the 
city’s afterschool administrative entity, currently the Philadelphia Health Management 
Corporation. The potential exists to link both PCAPS and data from the School District 
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of Philadelphia with the city’s developing data warehouse, CARES, which includes youth 
and family case management information from Philadelphia’s social service agencies. 

• In Antioch, Calif., the Youth Intervention Network partnered with AJW, Inc., and 
the city to conduct a study of youth ages 13-18 using cross-agency data to identify at- 
risk students and provide a variety of wraparound services. Youth truancy and student 
disengagement were the top two indicators of likelihood to commit or become victims 
of violence. Among the results of this initiative was a 79 percent reduction in truancy. 

NLC will provide a number of resources on integrated data systems throughout 2012 
and through our website at www.nlc.org/iyef. For a useful overview of this topic, see the 
publication, Connecting the Dots: The Promise of Integrated Data Systems for Policy Analysis 
and Systems Reform , developed in 2010 by the Intelligence for Social Policy initiative at the 
University of Pennsylvania at www.ispc.upenn.edu. 


CITY EXAMPLE: 


HARTFORD CONNECTS AT-RISK YOUTH 
WITH SUMMER EMPLOYMENT 

The Hartford Peacebuilders initiative is a 
"boots on the ground" violence intervention 
program that uses a risk assessment to 
identify young people who are likely to either 
perpetrate or be a victim of violence. These 
youth are, as closely as the programs that work in Hartford's neighborhoods can 
determine, the ones who are most likely to be catalysts for trouble - the "shot callers." 

Increasing the number of these youth who apply for and receive summer employment 
through the Capital Workforce Partners (the local workforce investment board) has 
been a multi-year priority for the city's Office of Youth Services. Both Capital Workforce 
Partners and the Peacebuilders are members of Hartford Connects, the city's multi- 
agency integrated data system. 

In 2010, the Office of Youth Services sent 400 applications for summer employment 
to the Peacebuilders to be distributed to the youth with whom they worked. That fall, 
city staff used the Hartford Connects database to match the Peacebuilders case files 
against the city's summer employment program records and estimate what percentage 
received a job. This was "great information to have," said Rachel Botts, the city's 
former program performance manager, and led to a focused conversation. "We need 
to improve this figure next year. We really need you guys to make a push. What are 
the barriers to your kids getting a job?" Botts asked her partners at Peacebuilders. 

There were some barriers, it turned out. In early 201 1, the Office of Youth Services 
hosted an "employability day" where they opened up city hall to local youth seeking 
work. Youth services staff negotiated with the Hartford Public Schools to have district 
staff present to print out transcripts. The Hartford Bureau of Vital Records modified its 
requirements so that as long as youth could bring a photocopy of their parent's ID with 
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their parent's signature on an application, they could receive a birth certificate and 
Social Security card. The youth on file with the Peacebuilders received a flash drive 
with these applications and resume templates months in advance, along with access 
to the summer jobs application several days in advance of the rest of the community. 

That fall, the Office of Youth Services again debriefed the Peacebuilders. This time, 
they used Hartford Connects to gather information on the success of each individual 
Peacebuilders case to analyze how many of the initiative's youth were eligible for 
summer employment, how many applied, and how many of the applications were 
complete - by name and by case worker. The information helped them answer 
questions about what challenges affecting those specific youth made it so difficult to 
get them a summer job and how next year could be different. 

Moving forward, the Office of Youth Services, now integrated into Hartford's 
Department of Families, Children, Youth, and Recreation, is working with Capital 
Workforce Partners to go even a step further: to notify Peacebuilders every time one of 
the youth on their case files submits an incomplete job application, while there is still 
time to fix the problem. 
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SECTION 3: 


DATA STEWARDSHIP: 
HOW TO PROTECT AND 
SHARE INFORMATION 


entities and schools 


A fterschool management information systems process and store a tremendous amount 
of information on youth participants. Among the crucial responsibilities of an MI 
administrator is to assure students, parents, and each of the project partners that he 
or she is a responsible steward of this sensitive information. In practice, this means providing 

guarantees that the data in the system are “fit for purpose” and that 
Chart 5. Information sharing every reasonable precaution has been taken to prevent their misuse, 
between city coordinating 

The specific permissions and prohibitions governing the collection 
and use of data by an MI system are defined by an overlapping 
web of federal and state laws, the most prominent of which in the 
context of afterschool programs is the Family Educational Rights 
and Privacy Act (FERPA). 1 Untangling the specific application 
of these laws is not always straightforward. 2 

However, city leaders undertaking this effort should be 
encouraged by the success of their peers: 67 percent of 
the cities surveyed by NLC in 2011 have established a 
framework for sharing information among youth service 
organizations and schools. Local officials may also be 
encouraged by the recent FERPA rules issued by the U.S. 
Department of Education (described on page 45), which 
responded to some of the most common objections to expanding 
data sharing relationships with local education agencies. Cities 
that can demonstrate a commitment to the principles of 
fair information practices, that are familiar with how these 
Among the cities surveyed by NLC in practices have been embedded in federal law, and that are 
2011, two-thirds of city coordinating committed to building trust with school and other information 

entities share information with partners can expect to be successful in negotiating access to 
public schools. the data necessary to support their work. 



i 


Depending on the type of youth information being shared and how it would be used, other federal laws such as the Health Insurance 
Portability and Accountability Act (HIPAA) may apply For a much more detailed description of the interaction of state and federal 
privacy laws, see the review by Professor John Petrila referenced on page 48 and available through the Intelligence for Social Policy 
website at www.ispc.upenn.edu. 


2 Though the legal framework for information privacy is complex and changing, it is grounded in a set of international principles that 
have remained fairly consistent for 40 years and which provide the foundation of privacy law in the United States. These principles are 
included in Appendix A. 
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PRIVACY 


The Family Educational Rights and Privacy Act (FERPA), section 444 of the federal 
General Education Provisions Act (GEPA), governs the confidentiality and permitted uses 
of educational records. In the absence of a more restrictive state statute, this laws provisions 
determine whether and with whom schools can share student information. FERPA applies 
to any recipient of funds from the U.S. Department of Education, including local and state 
education agencies but generally excluding private and parochial schools. 

Afterschool providers, city coordinating entities, and third-party program evaluators all 
fall outside of the list of organizations generally permitted access to student records by 
FERPA. 3 At least three strategies are available to cities seeking to work within or around this 
prohibition, however, depending on how the partners intend to use student information: 

1. partnering with schools to conduct afterschool evaluations; 

2. negotiating designation as an “agent” of the schools to access student information; and 

3. requesting prior written permission from each students parent or guardian to share 
academic information with providers. 

The first two strategies - school-based evaluation and designation as a FERPA-defined “agent” 
of the schools - are most useful for evaluating programs and overall youth outcomes. These 
strategies may, for example, allow for a comparison of youth participating in afterschool 
programs to the general student population and cohorts of non-participating youth. 

On the other hand, if the afterschool partnership would like to allow providers access to 
individual student data for purposes of case management on a day-to-day basis, prior written 
consent from each students parent or guardian is required. These three strategies are not 
mutually exclusive, and each is described in more detail below. 


1 . Have Schools Analyze Student Data (School-Based Evaluation) 

Schools may share information on youth outcomes if it is reported in such a way that no 
individual student’s performance can be determined (see page 48 for a list of information 
that can be shared). This stipulation permits schools to share data on the performance of 
their students by cohort, including by school, class, demographic characteristic, or - if 
they so choose - by participation in afterschool programming. Several citywide afterschool 
systems, including those in Grand Rapids, Mich., and Nashville, Tenn., are either funding 
or considering whether to fund a research position within the public schools to serve as 
a liaison to citywide afterschool partners and conduct this kind of analysis “from inside 
the FERPA firewall.” While this strategy can be very effective, it may not be feasible in 
cases where afterschool program participants attend multiple school districts or where those 
school districts are themselves resource-constrained. 


3 Individuals and organizations permitted by default to access student records include students, parents, school officials with a legitimate 
educational interest, schools to which a student is transferring, parties such as banks connected to student financial aid, accrediting 
organizations, and state and local authorities pursuant to state law or as related to a health emergency or judicial order. 
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2. Designate a City Coordinating Entity as an Agent of the Schools 
(Access to Student Records Without Consent) 

Federal regulations (CFR Title 34 § 99.31) define the conditions under which schools 
may release student records without the prior consent of parents or students, including for 
purposes of “audit, evaluation, or enforcement or compliance activity” related to education 
programs, including afterschool programs with an educational focus (see page 48). To quality 
for this exemption from FERPA, the recipient must qualify as an “authorized representative” 
of the schools, enter into a written agreement that governs the protection and use of the 
student data, and identify the local, state or federal law that calls for the audit, evaluation, or 
compliance activity. 4 

Authorized representatives can include independent consultants, university centers, or city 
coordinating entities, provided they have executed a written agreement with the agency 
that addresses the elements described below. Until recently, FERPA was interpreted as 
requiring the education agency to have “direct control” over those it authorized to have 
access to individual student information, and this interpretation limited researchers to 
being employees or contractors working onsite. A December 2011 rule issued by the 
U.S. Department of Education eliminated this requirement. 

The written agreement between the schools and their representatives must include, at a 
minimum: 

• Clear designation of the individual or entity being authorized; 

• A catalog of specific personally identifiable information (PII) to be disclosed; 

• Identification of the purpose for the which the FERPA exemption is being claimed 
and a description of the activities in sufficient detail to confirm that it is legitimate and 
could not be accomplished without the disclosure of PII; 

• A description of the purpose, scope, and duration of the study; 

• The terms under which PII will eventually be returned to the agency or destroyed by 
the representative, including a timeframe according to which this will be accomplished; 
and 

• Policies and procedures to ensure that PII is not intentionally or accidentally redisclosed 
or used for any purpose not explicitly permitted in this written agreement. 


3. Obtain Permission from Parents (Prior Written Consent) 

To inform day-to-day decision making and to share academic information with afterschool 
providers about individual students, most citywide afterschool systems ask parents and 
guardians to provide written consent for schools to share information with afterschool 


Under this exception, FERPA does not forbid the sharing of student information without consent, but the local or state education agency 
will have to determine whether it has reason to undertake one of those activities. 
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providers. Cities generally renew this consent annually though there are no general 
prohibitions against requesting consent to share information for two or more years. 

Parents’ permission can be collected by providers during enrollment or by schools during 
student registration, and most MI systems will provide a “flag” for each individual student 
that indicates whether or not this permission has been granted. The consent form provided 
to parents and guardians should include a list of the specific PII that may be disclosed, should 
state the purpose of the disclosure, and should clearly identify the organizations (or class of 
organization) to whom the disclosure may be made. 


NEW FERPA RULEMAKING 


In December 201 1, the U.S. Department of Education amended the regulations gov- 
erning the implementation of FERPA, primarily to reduce perceived barriers to the ap- 
propriate sharing of information for educational purposes. Two changes are of signifi- 
cant interest to afterschool providers: 

1. The term "Education Program" is now explicitly defined to include 
most afterschool programs. This change removes the possible objection that 
afterschool programs would not qualify for an Evaluation Exemption (using strategy #2 
on page 46). 

The fine print: Education Programs are defined as any programs that are principally 
engaged in the provision of education, including, but not limited to, early childhood 
education, elementary and secondary education, postsecondary education, special 
education, job training, career and technical education, and adult education, regardless 
of whether the program is administered by an educational authority. The rulemaking 
explicitly includes educational programs conducted by correctional and juvenile justice 
facilitates, dropout prevention and recovery programs, afterschool programs dedicated 
to enhancing academic achievement, and high school equivalency programs, 
"regardless of where or by whom they are administered." However, the rulemaking 
excludes programs that are principally engaged in recreation and entertainment (so- 
called "gym and swim"). 5 

2. Authorized Representatives no longer need to be under the "direct 
control" (and supervision) of the schools. 

The fine print: Authorized Representatives are defined generally as any entities or 
individuals designated by a state or local educational authority or an agency headed 
by an official listed in CFR Title 34 § 99.31(a)(3) - the Secretary of Education, the 
Comptroller General of the United States, or the Attorney General of the United States 
- to conduct, with respect to federal or state-supported education programs, any audit, 
evaluation, or compliance or enforcement activity in connection with federal legal 
requirements related to those programs. The incorrect interpretation by many education 
agency legal counsels that this representative must be under the "direct control" of the 
authorizing agency and therefore limited to agency staff and direct contractors has 
been clarified by the recent FERPA rulemaking; there is no such requirement. 


5 Page 75614 of the Federal Register, Vol. 76, no. 232 issued Friday, December 2, 2011. http://www.gpo.gov/fdsys/pkg/FR-201 1-12-02/ 
pdf/201 l-30683.pdf 
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FREQUENTLY ASKED QUESTIONS AND FEDERAL PRIVACY RESOURCES 


FAQ: What is "directory information?" 

FERPA permits very general information - so-called "directory information" - about 
students to be shared without student consent. This information includes their name, 
address, phone number and email address, dates of attendance, degrees awarded, 
enrollment status and major field of study. Institutions must notify students that the 
release of this information is permitted and provide them with an opportunity to opt 
out of having their directory information shared. This directory information does not 
include information on student behavior or academic outcomes, however, and may be 
of little or no use in evaluating the effectiveness of afterschool programs. 

FAQ: What are "reasonable methods" to protect educational data? 

FERPA requires that education agencies take all "reasonable methods" to ensure 
that student information is protected and used by its agents only for specifically 
authorized purposes. The U.S. Department of Education declined to define these 
purposes exactly, but provided a list of best practices in Appendix A of the December 
201 1 rulemaking. These practices include obtaining assurances against redisclosure, 
setting clear expectations around data destruction, maintaining a right to audit, 
verifying the existence of a data security plan, and ensuring the existence of a data 
stewardship plan (clear internal policies and procedures). 

Further Privacy Resources: 

• Privacy and Technical Assistance Center (PTAC) 

In April 201 1, the U.S. Department of Education hired its first chief privacy 
officer (CPO), Kathleen Styles. The CPO heads PTAC, which offers a growing 
selection of technical papers and webinars on data privacy and security matters 
at http://www2.ed.gov/ptac. 

• Basic Concepts and Definitions for Privacy and Confidentiality 
in Student Education Records 

This technical brief from PTAC describes the basic concepts and legal framework 
governing the release of student information. Several model memoranda of 
understanding are available through the U.S. Department of Education website 
at http://www2.ed.gov/about/offices/list/ovae/pi/cte/uiferpa.html. 

• Legal Issues in the Use of Electronic Data Systems for 
Social Science Research 

Professor John Petrila of the University of South Florida provides an excellent 
overview of the laws and legal issues involved in sharing and using individual 
information, with an emphasis on research uses. Further information is available 
through the Intelligence for Social Policy project underway at the University of 
Pennsylvania at http://www.ispc.upenn.edu/. 
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SECURITY 


Security is the enforcement of a privacy agreement. The assurances made to students, parents, 
and data partners that their confidentiality will be protected are only as credible as the ability 
of the coordinating entity to enforce them, and the trust between local partners can be 
permanently broken by negligence, malfeasance or the unauthorized redisclosure of private 
information. Management information systems present a new set of risks, in this regard, but 
also provide a set of tools for managing these risks. 

Negligence is at the root of most security breaches. For example, in late 2011, the Wakulla 
County School District in Florida accidentally published the FCAT scores and Social Security 
numbers of 2,400 students to an open web server. Parents discovered the problem when one 
of them used Google to search for their child’s name. In another recent incident, boxes 
filled with student information, including applications for free and reduced price school 
meals that contained financial information, were left in the garbage by a cleaning crew at 
an elementary schools in Santa Maria, Calif. It is not uncommon for laptop computers and 
thumb drives filled with unencrypted student files to go missing. 

Malfeasance and the deliberate redisclosure of private information present further risks for 
a coordinating entity to manage. Website and network hacking attempts are frequently 
opportunistic attempts to exploit badly maintained technology, but MI systems may also 
be deliberately targeted. Last year, students hacked a school district’s administrative record 
system in Blairsville, Pa., and downloaded teachers’ addresses, salaries and Social Security 
numbers. Furthermore, deliberate redisclosure of student information by staff or any 
“authorized representative” of an education agency is a serious breach of the law that will 
be investigated by the U.S. Department of Education’s Family Policy Compliance Office 
(FPCO). A finding by FPCO that a researcher or city coordinating entity working with the 
schools redisclosed student information in violation of FERPA carries at least a five-year 
ban on the receipt of any further private student data. If a compliance manager with access 
to the MI system were to share individual student academic information with providers 
without having received permission from parents, for example, the resulting FPCO 
enforcement could forbid the schools from sharing information with the coordinating 
entity for half a decade. 

To avoid these problems and protect the city’s and coordinating entity’s reputations, city 
leaders often provide the following safeguards: 


Create a Security Policy and Implement Internal Controls 

City coordinating entities often begin by taking an inventory of all of the sensitive and 
private information that is, or will be, stored and processed by the organization. Providing 
ongoing training to staff will make them aware of what can and cannot be shared with 
whom and under what circumstances. Cities must also implement policies to reduce or 
eliminate negligence, such as ensuring that any student data emailed or kept on personal 
computers is encrypted. Finally, it is important to develop a protocol for handling breaches. 
This protocol should include clear guidance on how to identify the problem, who to inform, 
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and what information to share. This security policy should be periodically reviewed by the 
organizations board or data governance committee. 

Further resources on data security include: 

• Data Stewardship: Managing Personally Identifiable Information in Electronic Student 
Education Records , a publication of the National Center for Education Statistics that is 
available at http://nces. ed.gov/pubsearch/pubsinfo. asp?pubid=201 1602 

• The Data Security Policy of the United Way of Greater Rochester (N.Y.), which is 
available online at www.nlc.org/afterschoolmis 


SECURE EMAIL AND FILE TRANSFER 


System administrators often need to download, manipulate and distribute participant 
information for a variety of reasons: at the request of a program manager, to verify the 
completeness and accuracy of provider information, to develop grant proposals, or 
to report to funders. These spreadsheets - which may contain sensitive or confidential 
information - are sent in the clear (i.e., not encrypted) to colleagues and forwarded 
according to the discretion of the recipient(s). 

There are at least two solutions to this problem: 

• First, consider using secure email. Elizabeth Ramsay Marchese, manager of quality 
assurance for the United Way of Greater Rochester, suggests using a security tool 
that encrypts the entire email message and any attachments and prevents any 
third party from intercepting or later accessing the contents of the email. 

• Encrypt the data file before transferring it. When the Jacksonville Children's 
Commission requests student information from Duval County Public Schools, 
SAMIS Senior Manager Juan Ruiz encrypts and emails a spreadsheet of all of 
the student participants for which the commission is seeking information to his 
colleagues at the district. The document returned by the district is also encrypted. 
One freely available file encryption tool is the PGP algorithm available through 
www.openpgp.org. 

Youth information is much more secure in a management information system than it is 
on a computer desktop, on a flash drive, or attached to an email. Encrypting these data 
when it is away from "home" is a best practice. 


Require Security Assurances from all Ml System Vendors 

The first threshold for any MI system is whether it can provide different levels of access to 
information to different system users: administrators, agency heads, program managers, and 
site staff. The configuration of these “role-based” permissions will be slightly different in every 
organization. However, it is crucial that the permission levels are carefully defined and that 
the vendor can accommodate them. Vendors should provide evidence that their system is 
secure from electronic attack, including information on the facility that hosts their servers and 
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the most recent audit of their security systems. Ideally all data hosted by the MIS would be 
encrypted not only in transit - between the web server and the web browser - but also when 
it is “at rest” on disk. Finally the vendor should provide a disaster recovery policy that outlines 
its procedures with regard to data breaches, application failures, and natural disasters. 


Audit 


Schools and other organizations that agree to provide information to a citywide afterschool 
system may request that one or both of the coordinating entity and its MIS vendor undergo a 
security audit. If not, the city or the coordinating entity should consider contracting for this 
service anyway. The audit should include a review of the organizations internal controls (its 
security policy). It should also, ideally, include two types of penetration testing: one attempt 
to hack the database from outside of the network and a second attempt using a guest account 
to “escalate privileges” and access information outside the scope of that user role. Reputable 
MI system providers are extremely security conscious and they should welcome this scrutiny. 


A trem 

protec, 


Chambers of commerce and local nonprofits such as the United Way may be able to 
provide a recommendation for a good network security firm, many of which are regional. 
Credentials are not a guarantee of quality, but the number and proportion of the firms 
employees certified as Information Systems Security Professionals can be an indicator of 

quality. More important is the standard the firm will 
use to evaluate an organizations security precautions 
and whether they have expertise in the privacy and 
security laws relevant to your data (such as FERPA). 
The Privacy and Technical Assistance Centers 
(PTAC) Data Security Checklist can be a helpful 
resource for developing a data security plan and is 
available at http://www2.ed.gov/ policy/ gen/ guid / 
ptac/ checklist.html. 


emendous amount of 
protected student information 
changes hands now, passed 
directly between teachers, 
principals and program officers 
informally, and stored in a 
variety of electronic and physical 
settings without much thought 
to security. " 


In a final analysis, the growth of student data systems 

- and out-of-school time MI systems among them 

- is likely to protect student privacy more than 
endanger it. A tremendous amount of protected 
student information changes hands now, passed 
directly between teachers, principals and program 

officers informally, and stored in a variety of electronic and physical settings without much 
thought to security. Teachers and afterschool program managers have every reason to share 
information on the youth they both serve. Data security is a major concern to schools, 
however, as many districts ratchet down their control over student information and replace 
these “ad hoc” teacher-provider relationships with formal agreements that meet the standard 
of federal and state privacy laws. 
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CITY EXAMPLE: 


GOVERNING AND USING DATA 
IN GRAND RAPIDS 

Bel ieve2 Become (B2B) is a place-based 
strategy in Grand Rapids, Mich., to close the 
achievement gap for 15,000 young people 
that live in four relatively disadvantaged 
areas of the city, dubbed "hope zones" (for more information, visit http://www. 
believe2become.org/). 

Two years ago, the DeVos Foundation piloted a management information system 
provided by nFocus to support the initiative's summer learning programs. nFocus' 
KidTrax system helps to facilitate information exchange between the schools and 
community-based providers and provides a rich set of data on program participation 
and academic outcomes for evaluators (see page 32 for information on the B2B 
initiative's systems architecture). 

To keep the providers, the MIS vendor, the local data partner, public schools and the 
third-party evaluators on the same page, the DeVos Foundation's research director, 
Edwin Hernandez, oversees no fewer than three working groups, which meet weekly 
or bi-weekly: 

• A school-based committee that includes representatives from the Grand Rapids 
Public Schools (GPRS) and the Community Research Institute (CRI), the local 
data partner that links and de-identifies GPRS data for use by the B2B initiative 
and members of the evaluation team. 

• An evaluation committee that includes CRI and the initiative's third-party 
evaluators. 

• A management information committee that includes foundation staff, the MIS 
vendor (nFocus), and CRI, and that troubleshoots technical and provider issues. 
The DeVos Foundation is also encouraging the development of an MIS users 
peer group. 

The number, structure and composition of data governance groups will vary according 
to the partners at the table in each community and the specific goals of the afterschool 
initiative. The example of 
Grand Rapids provides : 
what Dr. Hernandez un- 
derstated ly describes as 
a "robust" approach to 
managing these informa- 
tional tasks. 


^PliPVP 




Security standards include the National Institute of Standards and Technology (NIST) guidelines, the International Standards 
Organization’s (ISO) 27001 framework on Information Security Management, and the National Security Agency’s (NSA) Information 
Assurance Directorate. 
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SECTION 4: 


SELECTING A 
COMMERCIAL MIS 

W hile a few notable citywide afterschool systems are supported by self-developed 
software, the majority of afterschool management information systems are built 
by a handful of commercial vendors. Most of these companies were incorporated 
in the 1990s, and they vary in size from having less than two dozen employees to more than 
100. The smallest company serves several hundred thousand children while the larger ones 
manage several million youth records. 

In January 2012, NLC issued a request for information (RFI) to eight MI system vendors 
asking for detailed product specifications in each of the approximately 10 domains listed below 
in the “MI System Feature Comparison” section. City leaders are encouraged to download the 
edited version of this RFI from www.nlc.org/afterschoolmis and adapt it as part of their own 
RFI/RFP process. 

In some respects, the products developed by these companies are similar to one another. Each 
is sold using a Software as a Service (Saas) model, with the software and data hosted offsite and 
made available to cities, providers, and programs through the Internet. This model frees clients 
from managing desktop software and servers, and it is scalable within the complex environment 
of a citywide afterschool system. Most of these products provide similar functionality around 
enrolling youth, tracking program attendance, and executing surveys, assessments, and reports. 

The companies surveyed by NLC for this report compete with one another not only on price, 
but also on the sophistication of the consultation, project management, and customer service 
they are able to offer to cities and other clients. The scope of their services differs, with some 
companies tailoring their product to an afterschool environment while others more easily 
incorporate information and work routines from other youth-serving agencies such as social 
services and workforce development. Some have domains in which they are particularly 
experienced, such as the transition from early education to school-aged afterschool, while 
others provide advanced functionality around grant management, systems integration, case 
work, reporting, or mobile applications. 


Disclaimer 

While the author has taken every effort to ensure this information is accurate as of early 
2012, these products are complex and fast-developing. NLC strongly encourages city leaders 
to contact these companies directly to discuss their specific needs and to verify the informa- 
tion presented in this report. NLC does not endorse any of these software applications. The 
“best fit” for a city or youth-serving agency depends on its specific business case, its budget, 
and the local expertise available to it. 
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SOFTWARE VENDORS [in alphabetical order] 


CITYSPAN TECHNOLOGIES, INC. 

Cityspan Technologies provides products tailored to the 
requirements of 21st Century Community Learning Centers, 
Supplemental Educational Services (SES), and Gear Up programs, 
as well as more general citywide and longitudinal data systems. 
The company’s software is often known under the product name 
Youthservices . net. 

Cityspan’s depth of experience with city leaders and school 
information systems is invaluable. The company is an accomplished 
back-end systems integrator and data warehouse manager. These 
features and Cityspan’s competitive price have led to its adoption 
by several of the country’s largest cities and school districts. 

Cities with sophisticated needs for control over the front-end 
interface and for complex in-line reporting flexibility should 
carefully evaluate whether and how Cityspan can meet these needs. 


^cityspan 


Web: www.cityspan.com 
Located: Berkeley, Calif. 
Founded: 1994 

For more information: 

Mark Min 
CEO 

(o) 510.898.2662 
mark@cityspan.com 


COMET INFORMATICS 

COMET is a more recent entrant to the market, 
having grown out of a partnership between the 
Children’s Institute in Rochester, N.Y., and the 
technology company SophiTEC. Despite its youth, 

COMET is a fairly comprehensive platform that 
supports longitudinal data warehousing and real- 
time information sharing at approximately 1,000 
sites managed by government agencies, community- 
based organizations and school districts. 

The Rochester Children’s Institute has contribut- 
ed its professional expertise to the development of 
COMET, particularly around the software’s use by 
programs serving pre-kindergarten and younger, 
school-aged children. COMET features a strong set 
of youth assessments, intelligently integrated into the 
MI system’s reporting routines, as well as some thoughtful approaches to tasks such as soliciting 
feedback from external stakeholders and sharing data with student information systems. 

COMET may not have the same depth or breadth of options for case management, roster 
of add-ons such as provider directories and advanced reporting features, or sufficient scale to 
provide the training options offered by several of the leading MI system providers. It may, 
however, be an innovative and competitive option for many applications. 




COMET* 


Web: www.comet4children.com 
Located: Pittsford, N.Y. 

Founded: 201 1 

For more information: 

Serge Lossa 
President 

(o) 585.385.2310 
serge@sophitec.com 
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COMMUNITY TECHKNOWLEGE 


“CTK” manages two hosted services: Apricot and the Community Impact Online Data 
Manager (CI-ODM). In 2013, the two will be merged into a single platform. 

Apricot is a flexible client and case management 
package for smaller organizations that includes a set of 
volunteer management, outcomes tracking, and donor 
management tools. 

CI-ODM is designed for funders such as United Way 
or city agencies, and includes the functionality of 
Apricot as well as a built-in grant management module 
that manages the application, review and reporting life 
cycle. 

Unlike the other five companies reviewed in this report, 

Community TechKnowledge prices CI-ODM accord- 
ing to the budget of the organization/initiative purchas- 
ing it, rather than the number of agencies and sites. 


/U | 1/ Community 
VSIiX TechKnowledge 

Tflcfinology w * i ft a H m a rl ■ 

Web: www.communitytech.net 
Located: Austin, Texas 
Founded: 1999 

For more information: 

Patrick Wicker 

Director of Sales & Marketing 
(o) 512.345.9090x120 
pwicker@communitytech.net 


NFOCUS SOLUTIONS 

The company nFocus recently changed the second word in its name from “Software” to 
“Solutions” to reflect the broad role it has taken in helping public sector organizations 

integrate and use data to drive better results. Its Community 
Server and Community Compass products have expanded 
nFocus’ capacity to support multi-agency sharing and online 
public directories, respectively, while the “Trax” software 
remains the company’s core product for managing enrollment 
and attendance at afterschool and other out-of-school time 
programs. 

nFocus is especially experienced and innovative around the 
mechanics of tracking program attendance: the software license 
includes a bar code scanner, and mobile apps and scanners are 
available for remote or under-resourced sites. nFocus is also 
experienced with negotiating data sharing arrangements with 
K-12 schools, and it is developing a growing range of solutions 
for payment, case management, and grant management. 

As with Social Solutions’ ETO platform (see below), cities 
are likely to pay a premium for this flexibility - especially in “up front” costs. Cities with 
sophisticated needs around case and grant management across an array of public agencies will 
need to assess nFocus’ capabilities in this area more carefully, while youth-serving organizations 
will appreciate the company’s very strong offerings in these areas. 


. ; nfocus 

SOLUTIONS 

bridging the gap 

Web: www.nfocus.com 
Located: Austin, Texas 
Founded: 1994 

For more information: 

Richard Wells 

Business Development Manager 
(o) 512.276.2070x103 
rwells@nFocus.com 
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SOCIAL SOLUTIONS (EFFORTS TO OUTCOMES) 

Social Solutions is the largest of the companies reviewed in this 
report and its products serve the broadest range of agencies. 
The “Efforts to Outcomes (ETO)” platform supports youth- 
serving programs, workforce agencies, as well as public and 
private organizations in the areas of housing, health, and 
education. 

ETO includes case management tools as a central feature of 
the software’s approach to linking program activities to results, 
part of Social Solutions’ implementation of the “Results-Based 
Accountability (RBA)” framework. The software’s very well- 
developed reporting framework features an option to use the 
new RBA Scorecard product, business intelligence tools, and 
to integrate ETO data into Microsoft Live Office documents. 
ETO is a flexible product, and it includes a great number of 
template tools and resources made available through its Perform Well partnership as well as 
configurability to support tiered citywide partnerships across agencies. 

As might be expected, all of this functionality does not come cheap. Cities with narrower needs 
may find that a system that does less is a more appropriate and easily implemented match, 
while those developing broad multi-agency partnerships will appreciate ETO’s extensibility. 


Social Solutions ffl 

transforming human wrvices G WF 

Web: www.socialsolutions.com 
Located: Baltimore, Md. 
Founded: 1999 

For more information: 

Adrian Bordone 
Co-Founder 
o: 443.460.3383 
adrian@socialsolutions.com 


THOMASKELLY SOFTWARE 

ThomasKelly Software provides a suite of web-based solutions to manage registration, 
attendance, evaluation and reporting for afterschool and extended care programs. Their 
afterschool solution meets the requirements for grant-funded programs, including the 
21st Century Community Learning Centers. 

The extended care solution includes billing and 
payment features for summer camps and extended 
care. These two solutions can be bridged to offer 
a comprehensive solution for a city that has grant- 
funded as well as fee-based programs. 

The company’s leading product, EZReports, has 
a workflow that is very well-tailored to the grant 
management and evaluation tasks associated with 
these programs, with a clean interface and well- 
customized reports. While ThomasKelly Software 
has adapted this workflow to support several 
municipal customers, city leaders may wish to 
carefully evaluate whether the workflow meets their 
needs as ably as it does those of state clients. 


EZReports 

web-bisad Afteilschool Reporting System 

Web: www.thomaskelly.com 
Located: Houston, Texas 
Founded: 1996 

For more information: 

Felix Thomas 
President 

(o) 281.565.1460 
sales@thomaskelly.com 
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Other Ml Systems 


Of the commercial MI systems in use by cities surveyed by NLC for this report, two opted 
not to submit a full response to our request for information. 


Cayen Systems 

www.cayen.net 

CiviCore 

www.civicore.com 

Cayen is among the larger commercial 
providers of online data management 
solutions for schools and nonprofit 
organizations. As with several of the 
other vendors surveyed by NLC, it offers 
specific products aimed at 21st Century 
Community Learning Centers, SES 
programs, and Gear Up. The company 
has extensive experience building data 
sharing relationships with schools and 
working with city leaders to implement 
community-wide information systems. 

CiviCore's products include community 
mapping applications, a giving platform, 
and client relationship management 
packages for national volunteer and 
mentorship programs. Though it is not 
strictly an afterschool Ml system provider, 
CiviCore is developing the software for 
Denver's citywide Community Partnership 
System, and it has both the platform 
and evaluation and systems integration 
experience to build custom solutions for 
other cities with specific needs. 


Ml SYSTEM FEATURE COMPARISON 


City leaders should, of course, consult their own requirements when soliciting proposals and 
exercise due diligence in verifying the accuracy of each product s claims. This report provides 
a high-level starting point for that comparison, but it does not describe the nuances of each 
MI systems features and implementation - and those nuances are important. 


Enrollment, Attendance and Participation 


Take attendance using rosters, PCs and mobile devices 
Track youth participation by activity 
Group students, guardians together as "households" 
Integrate with card readers and scanners 
Allow batch import from spreadsheets/ other applications 


Cityspan 

COMET 

ETO 

Cl ODM 

KidTrax 

EZReports 

/ 

/ 

/ 

/ 

' 1 


/ 

/ 

/ 

/ 

/ 

/ 

/ 

/ 

/ 

/ 

/ 

/ 

/ 

/ 

/ 

CUSTOM 

/ 

/ 

/ 

/ 

/ 

/ 

/ 



All of these products have similar “basic” attendance functions: they can print attendance 
rosters, accept input from personal computers and mobile devices, and can track student 
attendance by activity so that cities can conduct more detailed analyses of youth participation. 
Each of them can also group students together with parents and guardians to form households 
for communications and reporting purposes. 

The functionality of these MI systems varies when it comes to how closely they integrate 
with scanners and swipe card systems and to what degree they allow the batch import from 
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spreadsheets and other software solutions. Several support only selected web browsers and 
operating systems, generally requiring later versions of Internet Explorer on Windows platforms. 

Advanced features available from some MI systems include the support of electronic 
signatures (for logging participation in billable activities), custom-built apps for smartphones 
and tablets, and mobile hand-held scanners for remote or computer-free facilities. 


Case Management 


Record individual youth service plans and set milestones 
Track interactions between staff and youth/ clients 
Facilitate referrals among organizations that use MIS 
Facilitate referrals to external organizations 


Cityspan 

COMET 

ETO 

Cl ODM 

KidTrax 

EZReports 


CUSTOM 

/ 

/ 

/ 

/ 

/ 

/ 

V 

V 

V 

V 

/ 


/ 

V 

/ 


/ 


V 

V 

V 



For MI systems that offer case management functionality, the ability to track youth-staff 
interactions, file notes on individual youth, and create youth service plans and milestones 
are standard issue, though the sophistication of how these functions are implemented can 
vary greatly. Cityspan charges a nominal setup and project management fee to enable case 
management, while nFocus charges a per-site license and maintenance service agreement 
(MSA) fee. ETO s “Point of Service” attendance and case management tool is included in 
the cost of the product. 

While all of the systems that support case management include a “referral” function, only 
three support referrals to organizations outside of the MI system, generally by email. These 
referral routines vary in sophistication, particularly in how they “close the loop” on whether 
youth referred for service are accepted and treated. In some cases, companies can build in a 
“ticketing” system for issuing and tracking referrals to external organizations. 


Agency, Site and Staff Management 


Record organizational characteristics (e.g., location, quality) 

Track staff credentials 
Associate staff with specific classes and program activities 
Can publish online (public) provider directory 

All of the systems can be configured to record organizational characteristics (such as program 
quality, location, services, and languages spoken), to track detailed information on staff 
qualifications, and to associate staff with specific classes and program activities. Again, there 
is wide diversity in how each of these functions is accomplished that will serve some city 
applications better than others. Measuring staff efforts toward outcomes is central to Social 
Solutions’ approach and workflow; these functions are built into ETO and into Cityspan, 
COMET, Cl ODM and EZReports to varying degrees. nFocus offers a full-featured module, 
StafITrax, for an additional per-site license fee and MSA. 


Cityspan COMET ETO Cl ODM KidTrax EZReports 
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Three of the six providers can publish online program directories. Of these, the nFocus 
Community Compass is the most developed product. All three companies charge an 
additional fee for this service. 


Survey and Assessment Instruments 


Administer surveys and outcome assessments 
Online distribution to non-participants (parents, teachers) 
Responses keyed to specific recipients 
Administrators can create custom surveys and assessments 


Cityspan 

COMET 

ETO 

Cl ODM 

KidTrax 

EZReports 

/ 

/ 

/ 

/ 

/ 
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/ 
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/ 
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Surveys and assessments are important for estimating youth development outcomes and 
program quality over time. For example, one set of instruments is closely associated with the 
requirements of the 21st Century Community Learning Centers and SES funding sources, 
and every afterschool MI system surveyed by NLC offers tools that meet these federal reporting 
requirements. Extensive libraries of additional assessments are available free of charge or for a 
nominal fee from many of these vendors, though some instruments must be licensed from their 
owner (such as the Developmental Assets Profile from the Search Institute). The size of these 
libraries and the type of instruments they contain vary significantly between vendors and may 
be an important consideration for some cities as their use will likely speed implementation, 
reduce startup costs, align collective effort, and leverage field-based best practices. 

nFocus, COMET, Cl ODM and ETO each allow administrators to build their own 
instruments, though sometimes complex validation requirements and data interdependencies 
will obligate the customer to pay for additional help. In most cases, surveys can be distributed 
electronically. COMET, for example, provides a simple routine to send surveys to batches of 
parents or teachers to solicit feedback. These surveys can be anonymous or keyed to individual 
recipients, and each response is entered automatically into the database. 


Reporting 


Administrators can modify existing reports with new filters, 

elements 

System administrators can create and layout new reports 
Integrate with business intelligence applications (e.g., SAP) 
Advanced reporting features 
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Reporting capabilities clearly distinguish several of these MI systems from one another. Cities 
may wish to consider several factors when evaluating their options, including: 

• The relevance and ease-of-use of built-in report templates and filters: Though all of 
these packages are at least modestly configurable, some are weighted toward reporting 
on attendance, others toward youth development outcome measures relevant to federal 
programs, and still others toward agency performance management tasks. Cities may 
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find that they are best served by companies whose orientation is similar to their own. 
All of these companies will create additional stock reports for an hourly fee that varies 
between approximately $125 and $250. 

• Flexibility: Some MI systems provide very limited ability to customize existing reports, 
such as selecting new filters or data elements, while others provide more sophisticated 
“report builders” that allow system administrators to design complicated queries and 
design print and web templates as they choose. Some firms allow clients live (ODBC - 
open database connectivity) access to their database to implement third-party business 
intelligence tools, while several require the client to download and transfer the data 
manually instead. A citys need for this flexibility may depend on how well suited the 
“canned” reports are for their purpose. 

• Advanced features: Several MI systems can provide cities with a customizable “data 
dashboard” (an easy-to-read, graphic representation of key performance indicators), 
sometimes as part of a built-in business intelligence suite such as SAP BusinessObjects 
or Izenda. The new performance management-oriented “Results Scorecard” integrates 
with products from both Social Solutions and nFocus. Social Solutions can also embed 
its reporting within Microsoft Live Office, and its ETO Analytics package offers a 
variety of statistical tools for measuring the significance of youth interventions. These 
advanced features are a wonderful tool for city leaders who can use them, but their 
presence or absence is generally not among the most important reasons to select one 
MI system over another. 

Integration with Other Data Systems 


Can import Excel, Access, XML, and text files 
API allows integration with third-party applications 
Experience negotiating access to data from K-12 schools 
Experience integrating health and human services data 

All of these applications can import and export data in one or more of the following formats: 
Microsoft Excel, Microsoft Access, XML and text files. Some can also export to PDF, Microsoft 
Word and HTML. Support for Excel, Access and XML is not uniform across products; city 
leaders may wish to check each product’s functionality against their specific needs. 

Several companies provide a public application programming interface (API) or “web service” 
that defines how other software packages can interact and share data with the MI system 
(without resorting to exporting and importing data files). Providing an API is a best practice 
and an important interoperability feature. Some products may, however, provide only one- 
way sharing. Clients of the nFocus Community Server, for example, can use its API to 
submit data to the MI system from other applications but cannot use the API to extract data. 

With the exception of Cl ODM, all of these companies have worked with K-12 school 
systems to develop data sharing arrangements. Some have negotiated just a few; others have 


Cityspan COMET ETO Cl ODM KidTrax EZReports 


/ 

/ 

/ 


/ 


/ 

mu 

/ 


/ 

mu 

/ 

/ 

/ 

in 

/ 

V 



/ 


/ 

■ill 


Section 4: Selecting a Commercial MIS 


60 


negotiated dozens. This breadth of experience is important because each projects purpose, 
set of partners, and information technology constraints is distinct, and MI system vendors 
provide a range of crucial consultation and problem solving services. 

The cost of implementing school-afterschool data sharing can vary tremendously. Each of 
the companies surveyed by NLC charges a one-time setup fee that can range from several 
thousand dollars to several tens of thousands of dollars. Recurring charges are similarly 
variable. While some vendors may provide a more competitively priced service than others, 
in this regard, the larger determinants of cost are (a) the projects specific business case, 
including whether it is designed to provide real-time information sharing or periodic data 
warehousing; (b) the amount of consultation involved in winning the approval of city and 
school district leaders and their attorneys; and (c) the technological limitations of the school 
information system. Most companies will provide some free consulting while developing a 
service contract to better estimate these costs. 

Among the six vendors, Social Solutions has the greatest experience integrating data from 
other administrative systems, including health, human services, and workforce development 
agencies, while the nFocus Community Server is a more recent market entrant with an 
impressive set of collaboration and data visualization features. Cityspan works with several 
of the country’s largest school systems. All three of these products are currently being used to 
support one or more Promise Neighborhoods. 


Training and Support 


Initial onsite training 
Phone support 
24/7 customer service 
Live chat/ instant messaging 
E-mail support 
Online user discussion forums 
Online knowledge base/ FAQ 
Free webinar series 
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Most MI system vendors assume one or more days of on-site training will be necessary for 
the administrators, agency leads, and site users at the outset of a new project, charged at a 
daily rate. Some may substitute interactive online webinars for a lower fee. 

All companies provide phone and email support during business hours, and Cl ODM offers 
24/7 customer service. Vendors may offer tickets for customers to track ongoing service 
issues and at least two, ETO and COMET, provide online portals for customers to track 
their service requests. Finally, all companies offer online help, several host an active user 
forum, and ETO, Cl ODM and KidTrax provide an ongoing series of free webinars. 
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Additional Functionality 


Cityspan COMET ETO Cl ODM KidTrax EZReports 

Support all major web browsers 
Provide tools for detecting and resolving duplicates 
Support longitudinal data tracking and reporting 
Will allow city clients to host their own data 

Three companies require at least some system users to have specific browser/operating system 
combinations - usually a later version of Internet Explorer on a Windows platform. This 
requirement is an important consideration for cities planning to extend a system to providers 
with older or non-Windows personal computers. 

Preventing, detecting, and merging duplicate entries is an important chore for every MI 
system manager, and city leaders should explore the tools that each of these companies offers 
to reduce the associated administrative burden. Several will use the student ID number from 
linked school systems, where available, to avoid this problem altogether. Most provide a 
configurable tool that flags possible duplicates during entry by staff. Several products also 
provide a “merge duplicate” function for site managers. 

nFocus is the only company to permit cities to host their own data, though most of their 
clients follow what has become industry standard practice and opt to store their information 
offsite. In practice, it is usually less expensive and more secure for cities to contract with their 
MIS vendors to offload this responsibility and access the application “in the cloud.” 



Grant and Contract Management 
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It is outside the scope of this report to review grant and contract management applications, 
which are often freestanding and provided separately from afterschool information systems. 
For some cities, however, the ability to integrate their review, funding, performance 
management and invoicing procedures is a significant priority. Three of these companies 
can provide this functionality, generally as an additional, linked product. For an excellent 
review of grant management systems, see Information Age Associates at http://www.iaa. 
com/ resources.html. 


Section 4: Selecting a Commercial MIS 


62 



Financial 


Cityspan COMET ETO Cl ODM KidTrax EZReports 

Track fees, payment, and billing history by participant 

Generate invoices 

Integrate with a payment gateway for processing credit cards 
Allow online registration by parents and youth 
Integrate with accounting and billing software 

ETO, Cl ODM, and nFocus can somewhat integrate with accounting packages through 
customizing data import and export routines. Several companies allow online registration for 
activities and, in the case of nFocus and EZReports, integration with merchant services for 
online payment. With the exception of COMET, all of these products can track customer 
payments and generate invoices. 



COST CALCULATOR 


The six companies that responded to NLC’s January 2012 request for information each 
provided cost estimates for a generalized afterschool system, including “fixed” setup costs, 
training, customization, and recurring licensing costs. NLC used these estimates to provide 
an interactive cost calculator online at www.nlc.org/afterschoolmis. 

Cities and other users can tailor their cost estimate according to the number of agencies 
and program sites their MIS is projected to serve, as well as by feature set. They may, for 
example, project the cost of sharing school data (grades, attendance and behavior) with 
providers through the MIS, creating a public program locator, tracking staff attendance and 
participation, or enabling case management and service referral functions through the MIS. 

These cost estimates are in no way binding on the companies that participated, but they offer 
cities a reasonably good idea of the range of the fixed and recurring prices they might expect 
to pay for a citywide MI system, given certain parameters. 

Some important factors to consider when negotiating a price with a commercial vendor include: 

• Most companies charge “per site.” Companies may base their licensure fee on their 
client’s annual budget, number of partner agencies, afterschool sites, system users, or 
some combination of the four. Of these, the “per site” charge is the most common, and 
rates between the cheapest and most expensive varied nearly 200 percent. 

• The cost of sharing information with schools can vary tremendously. The cost to 
negotiate and implement an information sharing relationship with schools is extremely 
difficult to predict and depends in large part on the cooperation and technical 
sophistication of the school district and its student information system. Under the 
best of circumstances, some sort of regular data exchange can be established for several 
thousand dollars. However, quotes of several tens of thousands of dollars to build and 
sustain this link were common. 
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• Recurring costs matter more than startup costs. Using NLC s model, MIS 
implementation, customization and training costs averaged less than 20 percent of the 
total five-year cost of one of these systems. While more complex systems will require 
a greater investment on the front end, the bulk of long-term costs to a city and its 
coordinating entity are a function of recurring licensure fees. 

Cities should conduct an RFP process whenever feasible and to ask for detailed project 
budgets from applicants. The model used by NLC for this report is available online at www. 
nlc.org/ afterschoolmis. 
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APPENDIX A: FAIR INFORMATION 
PRACTICE PRINCIPLES (FIPPS) 


Information privacy is grounded in a set of international principles that have remained fairly 
consistent for 40 years and that provide the foundation of privacy law in the United States. 
As endorsed by the White House in 201 1, they include: 

Transparency: Organizations should be transparent and notify individuals regarding col- 
lection, use, dissemination, and maintenance of personally identifiable information (PII). 

Individual Participation: Organizations should involve the individual in the process of 
using PII and, to the extent practicable, seek individual consent for the collection, use, 
dissemination, and maintenance of PII. Organizations should also provide mechanisms 
for appropriate access, correction, and redress regarding use of PII. 

Purpose Specification: Organizations should specifically articulate the authority that 
permits the collection of PII and specifically articulate the purpose or purposes for which 
the PII is intended to be used. 

Data Minimization: Organizations should only collect PII that is directly relevant and 
necessary to accomplish the specified purpose(s) and only retain PII for as long as is 
necessary to fulfill the specified purpose(s). 

Use Limitation: Organizations should use PII solely for the purpose(s) specified in the 
notice. Sharing PII should be for a purpose compatible with the purpose for which the 
PII was collected. 

Data Quality and Integrity: Organizations should, to the extent practicable, ensure that 
PII is accurate, relevant, timely, and complete. 

Security: Organizations should protect PII (in all media) through appropriate security 
safeguards against risks such as loss, unauthorized access or use, destruction, modification, 
or unintended or inappropriate disclosure. 

Accountability and Auditing: Organizations should be accountable for complying with 
these principles, providing training to all employees and contractors who use PII, and 
auditing the actual use of PII to demonstrate compliance with these principles and all 
applicable privacy protection requirements. 

The FIPPs can be understood as an affirmative case for the protection of individual privacy and 
a description of the obligations of any organization that collects and stores this information. 
Federal and state privacy laws represent a minimum set of protections, not the limit of what 
city afterschool partners should do to protect student privacy. 

For more information, see Robert Gellmans review, Fair Information Practices: A Brief 
History , at http://bobgellman.com/rg-docs/rg-FIPShistory.pdf. 
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